Home Page
  • August 29, 2025, 01:32:34 pm *
  • Welcome, Guest
Please login or register.

Login with username, password and session length
Advanced search  

News:

Official site launch very soon, hurrah!


Pages: 1 [2] 3 4 ... 10
 11 
 on: November 20, 2024, 09:58:02 pm 
Started by Dakusan - Last post by Dakusan

I added a live “total progress” progress bar to the MD5Sum List Script. It utilizes a temp fifo file and a backgrounded “pv”.


 12 
 on: October 19, 2024, 05:22:25 am 
Started by Dakusan - Last post by Dakusan

I wanted to make a script to check if updated flatpak app permissions have changed before updating them. I decided to use ChatGPTs newest model, 01-preview for this task and holy crap am I impressed. I had to give it 14 commands to get to the final product and it seems to work great. That number could have been reduced quite a bit had I done things a bit differently.

I still had to find the problems, look up reference docs to correct it, and even debug in an IDE a little. But just telling it where the problems were, it got there in the end, and its user interaction output is way better than what I was planning on doing.

Sections:

My final updated copy of the script

<?php
// -- This script checks for Flatpak updates and reports permission changes. --

// Function to parse permissions from Flatpak metadata
function parsePermissions($content) {
   $permissions = [];
   $lines = is_array($content) ? $content : explode("\n", $content);
   $currentSection = '';
   $skipSections = ['Application']; // Sections to skip

   foreach ($lines as $line) {
       $line = trim($line);

       if (empty($line)) {
           continue;
       }

       // Check for section headers
       if (preg_match('/^\[(.*)\]$/', $line, $matches)) {
           $currentSection = $matches[1];

           // Skip the [Application] section
           if (in_array($currentSection, $skipSections)) {
               $currentSection = '';
               continue;
           }

           $permissions[$currentSection] = [];
       } elseif ($currentSection !== '') {
           // Only process lines within non-skipped sections
           $parts = explode('=', $line, 2);
           if (count($parts) == 2) {
               $key = $parts[0];
               $values = explode(';', trim($parts[1], ';'));
               $permissions[$currentSection][$key] = $values;
           } else {
               // Handle keys without '=' (e.g., single permissions)
               $permissions[$currentSection][$line] = [];
           }
       }
   }

   return $permissions;
}

// Function to compare permissions
function comparePermissions($current, $new) {
   $differences = [];

   // Get all sections
   $sections = array_unique(array_merge(array_keys($current), array_keys($new)));

   foreach ($sections as $section) {
       $currentSection = isset($current[$section]) ? $current[$section] : [];
       $newSection = isset($new[$section]) ? $new[$section] : [];

       // Get all keys in this section
       $keys = array_unique(array_merge(array_keys($currentSection), array_keys($newSection)));

       foreach ($keys as $key) {
           $currentValues = isset($currentSection[$key]) ? $currentSection[$key] : [];
           $newValues = isset($newSection[$key]) ? $newSection[$key] : [];

           // Compare values
           $added = array_diff($newValues, $currentValues);
           $removed = array_diff($currentValues, $newValues);

           if (!empty($added) || !empty($removed)) {
               $differences[$section][$key] = [
                   'added' => $added,
                   'removed' => $removed,
               ];
           }
       }
   }

   return $differences;
}

// Function to output a line with appId (if given), colored red (IsError=true) or green (IsError=false)
function outputColor($appId, $str, $isError=true, $NoNewline=false)
{
   // Determine if coloring should be used
   static $hasColors=null;
   if(!isset($hasColors)) {
       $hasColors=stream_isatty(STDOUT) && (!trim(`command -v tput`) || intval(`tput colors`)>=16);
   }

   echo
       (!$hasColors ? '' : ($isError ? "\e[31m" : "\e[32m")).
       ($appId ? "$appId: " : '').
       $str.
       (!$hasColors ? '' : "\e[0m").
       ($NoNewline ? '' : "\n");
}

// Function to display permission differences
function displayDifferences($appId, $differences) {
   outputColor($appId, 'Permission changes:');
   foreach ($differences as $section => $keys) {
       outputColor(null, "  [$section]");
       foreach ($keys as $key => $changes) {
           if (!empty($changes['added'])) {
               outputColor(null, "    + $key = " . implode(';', $changes['added']));
           }
           if (!empty($changes['removed'])) {
               outputColor(null, "    - $key = " . implode(';', $changes['removed']));
           }
       }
   }
   echo "\n";
}

// Function to ask the user a question
function askQuestion($Question) {
   outputColor('', "$Question (y/N): ", true, true);
   $handle = fopen('php://stdin', 'r');
   $line = fgets($handle);
   $answer = trim(strtolower($line));
   fclose($handle);
   return ($answer == 'y' || $answer == 'yes');
}

// Get architecture (moved above the loop)
$archCommand = 'flatpak --default-arch';
exec($archCommand, $archOutput);
$arch = trim(implode('', $archOutput));

// Step 1: Get the list of installed Flatpaks
$installedCommand = 'flatpak list -a --columns=application,branch,origin,options';
exec($installedCommand, $installedOutput);

// Remove header line if present
if (!empty($installedOutput) && strpos($installedOutput[0], 'Application ID') !== false) {
   array_shift($installedOutput);
}

// Build an associative array of installed applications
$installedApps = [];

foreach ($installedOutput as $line) {
   // The output is tab-delimited
   $columns = explode("\t", trim($line));
   if (count($columns) >= 4) {
       $appId = $columns[0];
       $branch = $columns[1];
       $origin = $columns[2];
       $options = $columns[3];

       $installedApps[$appId] = [
           'appId' => $appId,
           'branch' => $branch,
           'origin' => $origin,
           'options' => $options,
       ];
   }
}

// Get the list of updates
exec('echo n | flatpak update', $updatesOutput);
foreach ($updatesOutput as $Line) {
   if (preg_match('/^ *\d+\.\s+([\w\.-]+)/', $Line, $matches)) {
       $updatesAvailable[]=$matches[1];
   }
}

// Let the user confirm the updates
echo str_repeat('-', 80)."\n";
echo implode("\n", $updatesOutput)."\n";
echo str_repeat('-', 80)."\n";
if (empty($updatesAvailable)) {
   outputColor('', 'No updates available for installed Flatpaks.');
   exit(0);
}
if (!askQuestion('Found '.count($updatesAvailable).' updates. Continue?')) {
   echo "Updates canceled.\n";
   exit(0);
}

$permissionChanges = [];
foreach ($updatesAvailable as $appId) {
   if (!isset($installedApps[$appId])) {
       outputColor($appId, 'Installed app not found. Skipping.');
       continue;
   }

   $app = $installedApps[$appId];
   $branch = $app['branch'];
   $origin = $app['origin'];
   $options = $app['options'];

   // Determine if it's an app or runtime
   $isRuntime = strpos($options, 'runtime') !== false;

   // Paths to the metadata files
   if ($isRuntime) {
       $metadataPath = "/var/lib/flatpak/runtime/$appId/$arch/$branch/active/metadata";
   } else {
       $metadataPath = "/var/lib/flatpak/app/$appId/$arch/$branch/active/metadata";
   }

   // Check if the metadata file exists
   if (!file_exists($metadataPath)) {
       outputColor($appId, 'Metadata file not found. Skipping.');
       continue;
   }

   // Read current permissions from the metadata file
   $metadataContent = file_get_contents($metadataPath);
   $currentPermissions = parsePermissions($metadataContent);

   // Get new metadata from remote
   $ref = $appId . '/' . $arch . '/' . $branch;
   $remoteInfoCommand = 'flatpak remote-info --show-metadata ' . escapeshellarg($origin) . ' ' . escapeshellarg($ref);

   // Clear $remoteOutput before exec()
   $remoteOutput = [];
   exec($remoteInfoCommand, $remoteOutput);

   if (empty($remoteOutput)) {
       outputColor($appId, 'Failed to retrieve remote metadata. Skipping.');
       continue;
   }

   // Parse new permissions from the remote metadata
   $newPermissions = parsePermissions($remoteOutput);

   // Compare permissions
   $differences = comparePermissions($currentPermissions, $newPermissions);

   if (!empty($differences)) {
       $permissionChanges[$appId] = $differences;
       displayDifferences($appId, $differences);
   } else {
       outputColor($appId, 'No permission changes found.', false);
   }
}

// If there are no permission changes, inform the user
if (empty($permissionChanges)) {
   echo "No permission changes detected in the available updates.\n";
}

// Ask user if they want to proceed
if (askQuestion('Do you want to proceed with the updates?')) {
   // Proceed with updates
   echo "Updating Flatpaks...\n";
   passthru('flatpak update -y');
} else {
   echo "Updates canceled.\n";
}
?>
List of the changes I made from the AI produced script
  • Added “outputColor” function and wrapped appropriate echo statements into it
  • Added “-a” to “flatpak list”
  • Added output line “Installed app not found. Skipping”
  • Added output line “No permission changes found.”
  • Added “askQuestion” function since there are now multiple places in the script that ask questions
  • Changed get-updates command from “flatpak remote-ls --updates” to “echo n | flatpak update” (with a regex extraction). Also now confirms the update list with the user.
The commands I gave the AI

For demonstration purposes, I have left all of my original typos uncorrected below.


Write me a program in php that checks if there are any updates for my installed flatpaks. and then for each flatpak check and see if the permissions have changed between my current version and the new version. report to me the changes. then ask me if i want to continue with the updates.
your code doesnt work.
first $command = 'flatpak list --app --updates --columns=application,app,branch,origin,installation'; should be flatpak remote-ls --updates --columns=application,app,branch,origin (installation is an invalid column)
and then flatpak info --show-permissions doesnt give the original permissions. you have to get the original non-user-modified permissions from /var/lib/flatpak
the line if (empty($output)) { is incorrect, it should be if (count($output)) { as $output is an array, not a string
$columns = preg_split('/\s{2,}/', trim($line)); is incorrect. its tab delimited, so it just needs to be explode("\t", trim($line));
you also have a problem with the $metadataPath. its not always in /var/lib/flatpak/app/ . It may be in /var/lib/flatpak/runtime/. You can confirm which directory it should be in with the options column having "runtime" as an option.
in the --columns for your first command, "application" should be "name"
actually, you cant get the options from flatpak remote-ls --updates . you have to get the options from flatpak list
flatpak remote-info does not have an option --show-permissions
in your flatpak remote-info command, there is no --branch option.
there is no --assumeno option
there is no --dry-run permission. perhaps use --no-deploy?
you are incorrect. flatpak remote-info does not have an option --show-permissions . we already went over that. perhaps flatpak remote-info --show-metadata to get the permissions
flatpak remote-info does not have --branch
when checking the metadata, do not compare the info under [Application]
also, move the "Get architecture" line above its for loop
you have a problem with exec($remoteInfoCommand, $remoteOutput); you need to always empty $remoteOutput before running the exec. Otherwise, it appends the data to what is already there.


The final script the AI produced

<?php
/**
* This script checks for Flatpak updates and reports permission changes.
*/

// Function to parse permissions from Flatpak metadata
function parsePermissions($content) {
   $permissions = [];
   $lines = is_array($content) ? $content : explode("\n", $content);
   $currentSection = '';
   $skipSections = ['Application']; // Sections to skip

   foreach ($lines as $line) {
       $line = trim($line);

       if (empty($line)) {
           continue;
       }

       // Check for section headers
       if (preg_match('/^\[(.*)\]$/', $line, $matches)) {
           $currentSection = $matches[1];

           // Skip the [Application] section
           if (in_array($currentSection, $skipSections)) {
               $currentSection = '';
               continue;
           }

           $permissions[$currentSection] = [];
       } elseif ($currentSection !== '') {
           // Only process lines within non-skipped sections
           $parts = explode('=', $line, 2);
           if (count($parts) == 2) {
               $key = $parts[0];
               $values = explode(';', trim($parts[1], ';'));
               $permissions[$currentSection][$key] = $values;
           } else {
               // Handle keys without '=' (e.g., single permissions)
               $permissions[$currentSection][$line] = [];
           }
       }
   }

   return $permissions;
}

// Function to compare permissions
function comparePermissions($current, $new) {
   $differences = [];

   // Get all sections
   $sections = array_unique(array_merge(array_keys($current), array_keys($new)));

   foreach ($sections as $section) {
       $currentSection = isset($current[$section]) ? $current[$section] : [];
       $newSection = isset($new[$section]) ? $new[$section] : [];

       // Get all keys in this section
       $keys = array_unique(array_merge(array_keys($currentSection), array_keys($newSection)));

       foreach ($keys as $key) {
           $currentValues = isset($currentSection[$key]) ? $currentSection[$key] : [];
           $newValues = isset($newSection[$key]) ? $newSection[$key] : [];

           // Compare values
           $added = array_diff($newValues, $currentValues);
           $removed = array_diff($currentValues, $newValues);

           if (!empty($added) || !empty($removed)) {
               $differences[$section][$key] = [
                   'added' => $added,
                   'removed' => $removed,
               ];
           }
       }
   }

   return $differences;
}

// Function to display permission differences
function displayDifferences($appId, $differences) {
   echo "Permission changes for $appId:\n";
   foreach ($differences as $section => $keys) {
       echo "  [$section]\n";
       foreach ($keys as $key => $changes) {
           if (!empty($changes['added'])) {
               echo "    + $key = " . implode(';', $changes['added']) . "\n";
           }
           if (!empty($changes['removed'])) {
               echo "    - $key = " . implode(';', $changes['removed']) . "\n";
           }
       }
   }
   echo "\n";
}

// Get architecture (moved above the loop)
$archCommand = 'flatpak --default-arch';
exec($archCommand, $archOutput);
$arch = trim(implode('', $archOutput));

// Step 1: Get the list of installed Flatpaks
$installedCommand = 'flatpak list --columns=application,branch,origin,options';
exec($installedCommand, $installedOutput);

// Remove header line if present
if (!empty($installedOutput) && strpos($installedOutput[0], 'Application ID') !== false) {
   array_shift($installedOutput);
}

// Build an associative array of installed applications
$installedApps = [];

foreach ($installedOutput as $line) {
   // The output is tab-delimited
   $columns = explode("\t", trim($line));
   if (count($columns) >= 4) {
       $appId = $columns[0];
       $branch = $columns[1];
       $origin = $columns[2];
       $options = $columns[3];

       $installedApps[$appId] = [
           'appId' => $appId,
           'branch' => $branch,
           'origin' => $origin,
           'options' => $options,
       ];
   }
}

// Get the list of available updates
$updatesCommand = 'flatpak remote-ls --updates --columns=application';
exec($updatesCommand, $updatesOutput);

// Remove header line if present
if (!empty($updatesOutput) && strpos($updatesOutput[0], 'Application ID') !== false) {
   array_shift($updatesOutput);
}

// Build a list of applications that have updates
$updatesAvailable = array_map('trim', $updatesOutput);

if (empty($updatesAvailable)) {
   echo "No updates available for installed Flatpaks.\n";
   exit(0);
}

$permissionChanges = [];

foreach ($updatesAvailable as $appId) {
   if (!isset($installedApps[$appId])) {
       // The updated app is not in the installed apps list, skip it
       continue;
   }

   $app = $installedApps[$appId];
   $branch = $app['branch'];
   $origin = $app['origin'];
   $options = $app['options'];

   // Determine if it's an app or runtime
   $isRuntime = strpos($options, 'runtime') !== false;

   // Paths to the metadata files
   if ($isRuntime) {
       $metadataPath = "/var/lib/flatpak/runtime/$appId/$arch/$branch/active/metadata";
   } else {
       $metadataPath = "/var/lib/flatpak/app/$appId/$arch/$branch/active/metadata";
   }

   // Check if the metadata file exists
   if (!file_exists($metadataPath)) {
       echo "Metadata file not found for $appId. Skipping.\n";
       continue;
   }

   // Read current permissions from the metadata file
   $metadataContent = file_get_contents($metadataPath);
   $currentPermissions = parsePermissions($metadataContent);

   // Get new metadata from remote
   $ref = $appId . '/' . $arch . '/' . $branch;
   $remoteInfoCommand = 'flatpak remote-info --show-metadata ' . escapeshellarg($origin) . ' ' . escapeshellarg($ref);

   // Clear $remoteOutput before exec()
   $remoteOutput = [];
   exec($remoteInfoCommand, $remoteOutput);

   if (empty($remoteOutput)) {
       echo "Failed to retrieve remote metadata for $appId. Skipping.\n";
       continue;
   }

   // Parse new permissions from the remote metadata
   $newPermissions = parsePermissions($remoteOutput);

   // Compare permissions
   $differences = comparePermissions($currentPermissions, $newPermissions);

   if (!empty($differences)) {
       $permissionChanges[$appId] = $differences;
       displayDifferences($appId, $differences);
   }
}

// If there are no permission changes, inform the user
if (empty($permissionChanges)) {
   echo "No permission changes detected in the available updates.\n";
}

// Ask user if they want to proceed
echo "Do you want to proceed with the updates? (y/N): ";
$handle = fopen("php://stdin", "r");
$line = fgets($handle);
$answer = trim(strtolower($line));

if ($answer == 'y' || $answer == 'yes') {
   // Proceed with updates
   echo "Updating Flatpaks...\n";
   passthru('flatpak update -y');
} else {
   echo "Updates canceled.\n";
}

fclose($handle);
?>

 13 
 on: October 16, 2024, 11:51:56 pm 
Started by Dakusan - Last post by Dakusan
Adding a solution to my problem where movie posters kept disappearing in between plex server reboots.
For some reason it was storing the movie poster metadata in "Metadata/*/*/*/Upload" and it expected them in "Metadata/*/*/*/Contents/_combined/". Ran the following command in "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata" to fix the missing posters:
Code: [Select]
IFS=$'\n'; for i in `ls */*/* -d`; do cd "$i"; if [ -d "Uploads" ]; then echo "$i"; rsync -a Uploads/ Contents/_combined/; fi; cd -; done

 14 
 on: October 07, 2024, 11:15:50 pm 
Started by Dakusan - Last post by Dakusan
It's impossible to say without going through a lot of complex debugging steps with you, unfortunately. It's a very complex process with little fiddly bits. If you can find any error messages it would help.

 15 
 on: October 07, 2024, 12:22:10 am 
Started by Dakusan - Last post by Rauru
Hello Dakusan!
I have installed Firefox as a flatpak and KeepassXC as a regular .deb package. I followed your instructions to set it up but it looks like it's still not working.
Maybe I'm missing additional steps here? Could it be possible I need to change something on KeepassXC's side for the browser integration? Like a custom proxy location or custom browser config location?
Rauru

 16 
 on: October 02, 2024, 03:18:20 am 
Started by Dakusan - Last post by Dakusan

My Plex server decided to go through my database and screw with all the posters and art, so I made the below script which can be used to recover them. You should only use this if doing a full revert to backup is not a viable option. Do also note that if you have refreshed the metadata on any tv series since the backup, it could cause issues with the episode posters.




#Run the following commands as root
sudo su

#User variables
SERVICE_NAME='plexmediaserver'
BACKUP_LOCATION='/ROOT_BACKUP_LOCATION/'
PLEX_USER='plex'

#Derived variables (Slashes at end of directory paths are required)
SQLITE3_PATH="/usr/lib/$SERVICE_NAME/Plex SQLite"
DB_PATH="/var/lib/$SERVICE_NAME/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
IMG_PATH="/var/lib/$SERVICE_NAME/Library/Application Support/Plex Media Server/Metadata/"
CACHE_PATH="/var/lib/$SERVICE_NAME/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/"

#Stop the plex service until we have updated everything
echo "Stopping Plex"
systemctl stop "$SERVICE_NAME"

#Set the (user_thumb_url, user_art_url, and updated_at) from the old database in the new database.
#The updated_at is required to match the cached images
echo "Updating database"
sudo -u "$PLEX_USER" "$SQLITE3_PATH" "$DB_PATH" -cmd "ATTACH DATABASE '$BACKUP_LOCATION$DB_PATH' as OLDDB" "
UPDATE
 metadata_items AS NMDI
SET
 user_thumb_url=OMDI.user_thumb_url,
 user_art_url=OMDI.user_art_url,
 updated_at=OMDI.updated_at
FROM
   OLDDB.metadata_items AS OMDI
WHERE
   NMDI.id=OMDI.id
"

#Copy the posters and art from the old directory to the new directory
echo "Copying posters"
cd "$BACKUP_LOCATION$IMG_PATH"
find -type d \( -name "art" -o -name "posters" \) | xargs -n100 bash -c 'rsync -a --relative "$@" "'"$IMG_PATH"'"' _

#Copy the cached images over
echo "Copying cache"
rsync -a "$BACKUP_LOCATION$CACHE_PATH" "$CACHE_PATH"

#Restart plex
echo "Starting Plex"
systemctl start "$SERVICE_NAME"


 17 
 on: September 15, 2024, 10:24:45 pm 
Started by Dakusan - Last post by Dakusan
Original post for Some useful Plex commands can be found at https://www.castledragmire.com/Posts/Some_useful_Plex_commands.
Originally posted on: 09/16/24

The below commands are tailored for Linux, as file paths are different for Windows. They need to be ran with the sudo -u plex prefix for proper user access permissions.

Get list of libraries
'/usr/lib/plexmediaserver/Plex Media Scanner' --list

Force library intro dection
'/usr/lib/plexmediaserver/Plex Media Scanner' --analyze --manual --server-action intros --section LIBRARY_ID

Force library credit dection
'/usr/lib/plexmediaserver/Plex Media Scanner' --analyze --manual --server-action credits --section LIBRARY_ID

Export media list

sqlite3 '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db' '
SELECT MDI.id, file, title, MDI."index", hints, GROUP_CONCAT(T.text) AS taggings
FROM media_parts AS MP
INNER JOIN media_items AS MI ON MI.id=MP.media_item_id
INNER JOIN metadata_items AS MDI ON MDI.id=MI.metadata_item_id
LEFT JOIN taggings as T ON T.metadata_item_id=MDI.id AND T.text IN ("credits", "intro")
GROUP BY MDI.id
'
This outputs the following (column separated by pipe “|”) for each individual media item:
  1. Metadata item ID
  2. File path
  3. Title
  4. Episode Number
  5. Hints (e.x. =&episode=1&episodic=1&season=1&show=Ninja%20Turtles&show_name=Ninja%20Turtles&year=2002)
  6. “credits” and “intros” tags (if the credits and intros timestamps have been calculated)

 18 
 on: July 30, 2024, 04:46:45 pm 
Started by Dakusan - Last post by Dakusan

Yesterday I moved my Plex installation from a Windows machine to a Linux machine. The primary data folders that needed to be copied over were Media, Metadata, Plug-ins, and Plug-in Support. It doesn't hurt to copy over some of the folders in Cache too. It's possible there may be some more data that needs to be moved over, but I don't have documented what.

After moving all the data, I updated the paths in the database for the new machine. Doing this allowed me to keep everything as it was and no new refreshes/scans needed to be done.

The location of the Plex modified SQLite for me was at /usr/lib/plexmediaserver/Plex SQLite. So the following is the bash commands to stop the plex server and open SQLite editor on Linux Mint 21.3.


service plexmediaserver stop
/usr/lib/plexmediaserver/Plex\ SQLite '/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'

And the following is the SQL I used to replace a D: drive with a path of /home/plex/drives/d/. You can replace these strings in the below code with your custom drives and paths.


#Replace backslashes with forward slash in paths
UPDATE media_parts SET file=REPLACE(file, '\', '/');
UPDATE section_locations SET root_path=REPLACE(root_path, '\', '/');
UPDATE media_streams SET url=REPLACE(url, '\', '/') WHERE url LIKE 'file://%';

#Replace root paths
UPDATE media_parts SET file=REPLACE(file, 'D:/', '/home/plex/drives/d/');
UPDATE section_locations SET root_path=REPLACE(root_path, 'D:/', '/home/plex/drives/d/');
UPDATE media_streams SET url=REPLACE(url, 'file://D:/', 'file:///home/plex/drives/d/') WHERE url LIKE 'file://%';
UPDATE media_streams SET url=REPLACE(url, 'file:///D:/', 'file:///home/plex/drives/d/') WHERE url LIKE 'file://%';
UPDATE metadata_items SET guid=REPLACE(guid, 'file:///D:/', 'file:///home/plex/drives/d/');

 19 
 on: July 19, 2024, 03:57:03 am 
Started by Dakusan - Last post by Dakusan


I created this script because I wasn’t happy with the native Roboform importer inside Bitwarden. This fixed multiple problems including:
* Ignoring MatchUrls
* Parent folders weren’t created if they had no items in them (e.x. if I had identities in Financial/Banks, but nothing in /Financial, then it wouldn’t be created /Financial)
* Completely ignoring the extra fields (RfFieldsV2)


This fixes all those problems.


This needs to be ran via php in a command line interface, so `php convert.php`. It requires the [bitwarden cli “bw” command](https://bitwarden.com/help/cli/).
There are 2 optional arguments you can pass:
1) The name of the import file. If not given, `./RfExport.csv` will be used
2) The “bw” session token. If not given as a parameter, it can be set directly inside this file on line #4. You get this token by running `bw unlock` (after logging in with `bw login`).


This script does the following:
* Reads from a csv file exported by Roboform to import into bitwarden
* Runs `bw sync` before processing
* Imported Fields:
 * Name: Becomes the name of the item
 * Url: Becomes the URL for the item
 * MatchUrl: If this is not the same “Url” then it is added as another URL for the item
 * Login: Becomes the login (user) name
 * Pwd: Becomes the password
 * Note: Becomes the note
 * Folder: Item is put in this folder
 * RfFieldsV2 (multiple):
   * Fields that match the login or password fields are marked as “linked fields” to those. If the field names are the following, they are not added as linked fields: *user, username, login, email, userid, user id, user id$, user_email, login_email, user_login, password, passwd, password$, pass, user_password, login_password, pwd, loginpassword*
   * Fields that are different than the login or password are added as appropriate “Custom fields”. Supported types: '', rad, sel, txt, pwd, rck, chk, are
   * Each field has 5 values within it, and 3 of those seem to be different types of names. So as long as they are not duplicates of each other, each name is stored as a seperate field.
* If all fields are blank but “name” and “note” then the item is considered a “Secure Note”
* While reading the csv file for import, errors and warnings are sent to stdout
* After the csv import has complete, it will give a total number of warnings/errors and ask the user if they want to continue
* Creates missing folders (including parents that have no items in them)
* During export to bitwarden:
 * This process can be quite slow since each instance of running “bw” has to do a lot of work
 * Keeps an active count of items processed and the total number of items
 * If duplicates are found (same item name and folder) then the user is prompted for what they want to do. Which is either s=skip or o=overwrite. A capitol of those letters can be given to perform the same action on all subsequent duplicates.
 * Errors are sent to stdout


Download script here

<?php
global $BW_SESSION;
/** @noinspection SpellCheckingInspection,RedundantSuppression */
$BW_SESSION=($argv[2] ?? 'FILL_ME_IN');

print RunMe($argv)."\n";

//Make sure “bw” is installed, given session is valid, and is synced
function CheckBWStatus(): ?string
{
   if(is_string($StatusCheck=RunBW('status')))
      return $StatusCheck;
   else if(!is_object($StatusCheck) || !isset($StatusCheck->status))
      return 'bw return does not have status';
   else if($StatusCheck->status==='locked')
      return 'bw is locked';
   else if($StatusCheck->status!=='unlocked')
      return 'bw status is invalid';

   $ExpectedMessage='Syncing complete.';
   if(is_string($SyncCheck=RunBW('sync', null, false)))
      return $SyncCheck;
   else if($SyncCheck[0]!==$ExpectedMessage)
      return "Sync expected “${ExpectedMessage}” but got “${SyncCheck[0]}”";

   return null;
}

//Pull the known folders
function PullKnownFolders(): string|array
{
   $FolderIDs=[];
   if(is_string($FolderList=RunBW('list folders')))
      return $FolderList;
   foreach($FolderList as $Folder)
      $FolderIDs[$Folder->name]=$Folder->id;
   unset($FolderIDs['No Folder']);
   return $FolderIDs;
}

//Get the file handle and the column indexes
function GetFileAndColumns(): string|array
{
   //Prepare the import file for reading
   ini_set('default_charset', 'UTF-8');
   $FileName=$argv[1] ?? './RfExport.csv';
   if(!file_exists($FileName))
      return 'File not found: '.$FileName;
   else if(!($f=fopen($FileName, 'r')))
      return 'Error opening file: '.$FileName;

   //Check the header row values
   if(!($HeadRow=fgetcsv($f)))
      return 'Error opening file: '.$FileName;
   else if(!count($HeadRow) || $HeadRow[0]===NULL)
      return 'Missing head row: '.$FileName;
   if(str_starts_with($HeadRow[0], "\xEF\xBB\xBF")) //Remove UTF8 BOM
      $HeadRow[0]=substr($HeadRow[0], 3);
   unset($FileName);

   $ExpectedCols=array_flip(['Url', 'Name', 'MatchUrl', 'Login', 'Pwd', 'Note', 'Folder', 'RfFieldsV2']);
   $ColNums=[];
   foreach($HeadRow as $Index => $Name) {
      if(!isset($ExpectedCols[$Name]))
         if(isset($ColNums[$Name]))
            return 'Duplicate column title: '.$Name;
         else
            return 'Unknown column title: '.$Name;
      $ColNums[$Name]=$Index;
      unset($ExpectedCols[$Name]);
   }
   if(count($ExpectedCols))
      return 'Required columns not found: '.implode(', ', array_keys($ExpectedCols));
   else if($ColNums['RfFieldsV2']!==count($ColNums)-1)
      return 'RfFieldsV2 must be the last column';

   return [$f, $ColNums];
}

//Process the rows
function ProcessRows($f, array $ColNums, array &$FolderIDs): array
{
   $Counts=['Error'=>0, 'Warning'=>0, 'Success'=>0];
   $RowNum=0;
   $FolderNames=[];
   $Items=[];
   while($Line=fgetcsv($f, null, ",", "\"", "")) {
      //Process the row and add result type to counts
      $RowNum++;
      $Result=ProcessRow($Line, $ColNums);
      $Counts[$Result[0]]++;

      //Handle errors and warnings
      if($Result[0]!=='Success') {
            print "$Result[0]: Row #$RowNum $Result[1]\n";
            continue;
      } else if(isset($Result['Warnings']) && count($Result['Warnings'])) {
         print "Warning(s): Row #$RowNum ".implode("\n", $Result['Warnings'])."\n";
         $Counts['Warning']+=count($Result['Warnings']);
      }

      //Add the folder to the list of folders (strip leading slash)
      $FolderName=($Line[$ColNums['Folder']] ?? '');
      $FolderName=substr($FolderName, ($FolderName[0] ?? '')==='/' ? 1 : 0);
      $Result['Data']->folderId=&$FolderIDs[$FolderName];
      $FolderNames[$FolderName]=1;

      //Save the entry
      $Items[]=$Result['Data'];
   }
   fclose($f);
   return [$Counts, $FolderNames, $Items, $RowNum];
}

//Process a single row
function ProcessRow($Line, $ColNums): array
{
   //Skip blank lines
   if(!count($Line) || (count($Line)===1 && $Line[0]===null))
      return ['Warning', 'is blank'];

   //Extract the columns by name
   $C=(object)[];
   foreach($ColNums as $Name => $ColNum)
      $C->$Name=$Line[$ColNum] ?? '';

   //Check for errors and end processing early for notes
   if($C->Name==='')
      return ['Error', 'is missing a name'];
   else if($C->Url==='' && $C->MatchUrl==='' & $C->Login==='' && $C->Pwd==='') {
      return ['Success', 'Data'=>(object)['type'=>2, 'notes'=>$C->Note, 'name'=>$C->Name, 'secureNote'=>['type'=>0]]];
   }

   //Create a login card
   $Ret=(object)[
      'type'=>1,
      'name'=>$C->Name,
      'notes'=>$C->Note===''  ? null : $C->Note,
      'login'=>(object)[
         'uris'=>[(object)['uri'=>$C->Url]],
         'username'=>$C->Login,
         'password'=>$C->Pwd,
      ],
   ];
   if($C->MatchUrl!==$C->Url)
      $Ret->login->uris[]=(object)['uri'=>$C->MatchUrl];

   //Create error string for mist fields
   $i=0; //Declared up here so it can be used in $FormatError
   $FormatError=function($Format, ...$Args) use ($ColNums, &$i): string {
      return 'RfFieldsV2 #'.($i-$ColNums['RfFieldsV2']+1).' '.sprintf($Format, ...$Args);
   };

   //Handle misc/extra (RfFieldsV2) fields
   $Warnings=[];
   for($i=$ColNums['RfFieldsV2']; $i<count($Line); $i++) {
      //Pull in the parts of the misc field
      if(count($Parts=str_getcsv($Line[$i], ",", "\"", ""))!==5)
         return ['Error', $FormatError('is not in the correct format')];
      $Parts=(object)array_combine(['Name', 'Name3', 'Name2', 'Type', 'Value'], $Parts);
      if(!trim($Parts->Name))
         return ['Error', $FormatError('invalid blank name found')];

      //Figure out which “Names” to process
      $PartNames=[trim($Parts->Name)];
      foreach([$Parts->Name2, $Parts->Name3] as $NewName)
         if($NewName!=='' && !in_array($NewName, $PartNames))
            $PartNames[]=$NewName;

      //Process the different names
      foreach($PartNames as $PartName) {
         //Determined values for the item
         $LinkedID=null;
         $PartValue=$Parts->Value;
         /** @noinspection PhpUnusedLocalVariableInspection */ $Type=null; //Overwritten in all paths

         //Handle duplicate usernames and password fields
         if(($IsLogin=($PartValue===$C->Login)) || $PartValue===$C->Pwd) {
            if($Parts->Type!==($IsLogin ? 'txt' : 'pwd'))
               $Warnings[]=$FormatError('expected type “%s” but got “%s”', $IsLogin ? 'txt' : 'pwd', $Parts->Type);
            $Type=3;
            $PartValue=null;
            $LinkedID=($IsLogin ? 100 : 101);

            //If a common name then do not add the linked field
            /** @noinspection SpellCheckingInspection */
            if(in_array(
               strToLower($PartName),
               $IsLogin ?
                  ['user', 'username', 'login', 'email', 'userid', 'user id', 'user id$', 'user_email', 'login_email', 'user_login'] :
                  ['password', 'passwd', 'password$', 'pass', 'user_password', 'login_password', 'pwd', 'loginpassword']
            ))
               continue;
         } else {
            //Convert the type
            switch($Parts->Type) {
               case '': //For some reason some text fields have no type given
               case 'rad':
               case 'sel':
               case 'txt': $Type=0; break;
               case 'pwd': $Type=1; break;
               case 'rck': //Radio check?
                  $Type=0;
                  if($PartName!==$Parts->Name) //Ignore second names for radio checks
                     continue 2;
                  if(count($RadioParts=explode(':', $PartName))!==2)
                     return ['Error', $FormatError('radio name needs 2 parts separated by a colon')];
                  $PartName=$RadioParts[0];
                  $PartValue=$RadioParts[1];
                  break;
               case 'chk':
                  $Type=2;
                  if($PartValue==='*' || $PartValue==='1')
                     $PartValue='true';
                  else if($PartValue==='0')
                     $PartValue='false';
                  else
                     return ['Error', $FormatError('invalid value for chk type “%s”', $PartValue)];
                  break;
               case 'are': //This seems to be a captcha
                  continue 2;
               default:
                  return ['Error', $FormatError('invalid field type “%s”', $Parts->Type)];
            }
         }

         //Create the return object
         if(!isset($Ret->fields))
            $Ret->fields=[];
         $Ret->fields[]=(object)[
            'name'=>$PartName,
            'value'=>$PartValue,
            'type'=>$Type,
            'linkedId'=>$LinkedID,
         ];
      }
   }

   //Return finished item and warnings
   if(count($Warnings))
      $Warnings[0]="RfFieldsV2 warnings:\n".$Warnings[0];
   return ['Success', 'Data'=>$Ret, 'Warnings'=>$Warnings];
}

//Ask the user if they want to continue
function ConfirmContinue($Counts, $RowNum): ?string
{
   //Ask the user if they want to continue
   printf("Import from spreadsheet is finished. Total: %d; Successful: %d, Errors: %d, Warnings: %d\n", $RowNum, $Counts['Success'], $Counts['Error'], $Counts['Warning']);
   while(1) {
      print 'Do you wish to continue the export to bitwarden? (y/n): ';
      fflush(STDOUT);
      switch(trim(strtolower(fgets(STDIN)))) {
         case 'n':
            return 'Exiting';
         case 'y':
            break 2;
      }
   }

   return null;
}

//Get folder IDs and create parent folders for children
function GetFolders($FolderNames, &$FolderIDs): ?string
{
   foreach(array_keys($FolderNames) as $FolderName) {
      //Skip “No Folder”
      if($FolderName==='') {
         $FolderIDs[$FolderName]='';
         continue;
      }

      //Check each part of the folder tree to make sure it exists
      $FolderParts=explode('/', $FolderName);
      $CurPath='';
      foreach($FolderParts as $Index => $FolderPart) {
         $CurPath.=($Index>0 ? '/' : '').$FolderPart;

         //If folder is already cached then nothing to do
         if(isset($FolderIDs[$CurPath])) {
            continue;
         }

         //Create the folder
         print "Creating folder: $CurPath\n";
         if(is_string($FolderInfo=RunBW('create folder '.base64_encode(json_encode(['name'=>$CurPath])), 'create folder: '.$CurPath)))
            return $FolderInfo;
         else if (!isset($FolderInfo->id))
            return "bw folder create failed for “${CurPath}”: Return did not contain id";
         $FolderIDs[$CurPath]=$FolderInfo->id;
      }
   }
   return null;
}

//Pull the known items
function PullKnownItems($FoldersByID): string|array
{
   $CurItems=[];
   if(is_string($ItemList=RunBW('list items')))
      return $ItemList;
   foreach($ItemList as $Item) {
      $CurItems[($Item->folderId===null ? '' : $FoldersByID[$Item->folderId].'/').$Item->name]=$Item->id;
   }
   return $CurItems;
}

function StoreItems($Items, $CurItems, $FoldersByID): int
{
   print "\n"; //Give an extra newline before starting the processing
   $NumItems=count($Items);
   $NumErrors=0;
   $HandleDuplicatesAction=''; //The "always do this" action
   foreach($Items as $Index => $Item) {
      //Clear the current line and print the processing status
      printf("\rProcessing item #%d/%d", $Index+1, $NumItems);
      fflush(STDOUT);

      //If the item already exists then request what to do
      $FullItemName=($Item->folderId===null ? '' : $FoldersByID[$Item->folderId].'/').$Item->name;
      if(isset($CurItems[$FullItemName])) {
         //If no "always" action has been set in $HandleDuplicatesAction then ask the user what to do
         if(!($Action=$HandleDuplicatesAction)) {
            print "\n";
            while(1) {
               print "A duplicate at “${FullItemName}” was found. Choose your action (s=skip,o=overwrite,S=always skip,O=always overwrite):";
               fflush(STDOUT);
               $Val=trim(fgets(STDIN));
               if(in_array(strtolower($Val), ['s', 'o']))
                  break;
            }
            $Action=strtolower($Val);
            if($Val!==$Action) //Upper case sets the "always" state
               $HandleDuplicatesAction=$Action;
         }

         //Skip the item
         if($Action==='s')
            continue;

         //Overwrite the item
         if(is_string($Ret=RunBW('edit item '.$CurItems[$FullItemName].' '.base64_encode(json_encode($Item)), 'edit item: '.$FullItemName))) {
            $NumErrors++;
            printf("\nError on item #%d: %s\n", $Index+1, $Ret);
         }
         continue;
      }

      //Create the item
      if(is_string($Ret=RunBW('create item '.base64_encode(json_encode($Item)), 'create item: '.$FullItemName))) {
         $NumErrors++;
         printf("\nError on item #%d: %s\n", $Index+1, $Ret);
      }
   }
   return $NumErrors;
}

//Run a command through the “bw” application
function RunBW($Command, $Label=null, $DecodeJSON=true): string|object|array
{
   //$Label is set to $Command if not given
   if($Label===null)
      $Label=$Command;

   //Run the command and check the results
   global $BW_SESSION;
   exec(sprintf('BW_SESSION=%s bw %s --pretty 2>&1', $BW_SESSION, $Command), $Output, $ResultCode);
   if($ResultCode===127)
      return 'bw is not installed';
   else if($ResultCode===1)
      return "bw “${Label}” threw an error: \n".implode("\n", $Output);
   else if($ResultCode!==0)
      return "bw “${Label}” returned an invalid status code [$ResultCode]: \n".implode("\n", $Output);
   else if($Output[0]==='mac failed.')
      return 'Invalid session ID';
   else if(!$DecodeJSON)
      return [implode("\n", $Output)];
   else if(($JsonRet=json_decode(implode("\n", $Output)))===null)
      return "bw “${Label}” returned non-json result";

   //Return the json object
   return $JsonRet;
}

function RunMe($argv): string
{
   //Make sure “bw” is installed and given session is valid
   if(is_string($Ret=CheckBWStatus()))
      return $Ret;

   //Pull the known folders
   if(is_string($FolderIDs=PullKnownFolders()))
      return $FolderIDs;

   //Get the file handle and the column indexes
   if(is_string($Ret=GetFileAndColumns()))
      return $Ret;
   [$f, $ColNums]=$Ret;
   unset($argv);

   //Process the rows and ask the user if they want to continue
   [$Counts, $FolderNames, $Items, $RowNum]=ProcessRows($f, $ColNums, $FolderIDs);
   if(is_string($Ret=ConfirmContinue($Counts, $RowNum)))
      return $Ret;
   unset($Counts, $RowNum, $f, $ColNums);

   //Get folder IDs and create parent folders for children
   if(is_string($Ret=GetFolders($FolderNames, $FolderIDs)))
      return $Ret;
   unset($FolderNames);

   //Pull the known items
   $FoldersByID=array_flip($FolderIDs);
   if(is_string($CurItems=PullKnownItems($FoldersByID)))
      return $CurItems;

   //Store all items
   $FolderIDs['']=null; //Change empty folder to id=null for json insertions
   $NumErrors=StoreItems($Items, $CurItems, $FoldersByID);

   //Return completion information
   return "\nCompleted ".($NumErrors ? "with $NumErrors error(s)" : 'successfully');
}

 20 
 on: July 12, 2024, 10:36:53 pm 
Started by Dakusan - Last post by Dakusan

Since I have been moving my primary workstation to Linux, I have been in need of finding a new password manager after Roboform failed me. I decided to try out KeePassXC but was having trouble since my Ungoogled Chromium browser was sandboxed in Flatpak. The instructions for this on Unix Stack Exchange were no longer working so I had to dig into it myself.


The 2 primary problems were that:
1) The browsers’ sandboxes did not have the QT libs.
2) The interprocess communication pipe socket had been renamed from kpxc_server to org.keepassxc.KeePassXC.BrowserServer.


The following are the instructions to get KeePassXC running in Flatpak versions of both Chrome and Firefox. This was tested on Linux Mint 21.3 with both Ungoogled Chromium and Firefox. You will need to change the KP_FLATPAK_PACKAGE if you use other versions of Chrome.


  1. Run the relevant below environment variables in your command shell before running the commands in the following steps:

    #Shared environment variables:
    KP_CUSTOM=/home/$USER/keepass-browser
    KP_JSON_NAME=org.keepassxc.keepassxc_browser.json

    #Chrome environment variables:
    KP_FLATPAK_PACKAGE=io.github.ungoogled_software.ungoogled_chromium
    KP_JSON_START=~/.config/chromium/NativeMessagingHosts
    KP_JSON_END=~/.var/app/$KP_FLATPAK_PACKAGE/config/chromium/NativeMessagingHosts

    #Firefox environment variables:
    KP_FLATPAK_PACKAGE=org.mozilla.firefox
    KP_JSON_START=~/.mozilla/native-messaging-hosts
    KP_JSON_END=~/.var/app/$KP_FLATPAK_PACKAGE/.mozilla/native-messaging-hosts
          
  2. Install and enable the browser extension:
    KeePassXD > Tools > Settings > Browser Integration:
    • Check “Enable Browser Integration”
    • Check “Chromium” and/or “Firefox”
    • Download the plugin listed on this screen in your browser
    • Click "OK"
    Note: This creates $KP_JSON_START/$KP_JSON_NAME

  3. Set up the needed files in the sandbox:

    #Put KeePass proxy and needed library files in user directory
    mkdir -p $KP_CUSTOM/lib
    mkdir -p $KP_JSON_END #Needed for firefox
    cp /usr/bin/keepassxc-proxy $KP_CUSTOM/
    rsync -a /usr/lib/x86_64-linux-gnu/libicudata* /usr/lib/x86_64-linux-gnu/libicuuc* /usr/lib/x86_64-linux-gnu/libicui* /usr/lib/x86_64-linux-gnu/libdouble* /usr/lib/x86_64-linux-gnu/libsodium* /usr/lib/x86_64-linux-gnu/libQt5* $KP_CUSTOM/lib

    #Copy the JSON file to the Flatpak app directory and change the executable path in the file
    cp $KP_JSON_START/$KP_JSON_NAME $KP_JSON_END/
    sed -i "s/\/usr\/bin\//"$(echo $KP_CUSTOM | sed 's_/_\\/_g')"\//" $KP_JSON_END/$KP_JSON_NAME
          
  4. Add permissions to the Flatpak:

    flatpak override --user --filesystem=$KP_CUSTOM:ro $KP_FLATPAK_PACKAGE #Only required if home directory is not shared to the Flatpak
    flatpak override --user --filesystem=xdg-run/org.keepassxc.KeePassXC.BrowserServer:ro $KP_FLATPAK_PACKAGE
    flatpak override --user --env=LD_LIBRARY_PATH=$(flatpak info --show-permissions $KP_FLATPAK_PACKAGE | grep -oP '(?<=LD_LIBRARY_PATH=).*')";$KP_CUSTOM/lib" $KP_FLATPAK_PACKAGE
          

Pages: 1 [2] 3 4 ... 10