|
-
October 03, 2024, 06:53:46 pm
- Welcome, Guest
News:Official site launch very soon, hurrah!
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Messages - Dakusan
Pages: 1 2 [3] 4 5 ... 37
31
« on: July 07, 2017, 04:52:58 pm »
Most likely a database update  I'll take a look at it as soon as I have a chance. (Should be sometime in the next week)
32
« on: June 26, 2017, 02:35:11 am »
Ahah. Should have thought of that. The latest version on github has options for plex user accounts, but I don't think I've put it on my site yet.
33
« on: June 24, 2017, 11:47:49 am »
First, "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" needs to be quoted, since it has spaces, or your command line interpreter will think those are separate parameters.
I'm surprised it decided to import at all with what you gave it, since you gave an unquoted (and therefore bogus) path to the database. I'm guessing it found the path on its own so it didn't error out.
Second, you only gave the "Playlist_Path" parameter and not a "Plex_Playlist_Name" parameter.
34
« on: June 19, 2017, 04:09:42 pm »
"DB Error: no such module: fts4" indicates an outdated or incomplete sqlite3 version. Google also points out that this happens when trying to compile for iOS, or possibly osx.
35
« on: May 30, 2017, 08:18:38 pm »
To log all process spawns from a user:
(Fill in USERNAME)
auditctl -a exit,always -S execve -F uid=USERNAME
To grep for only these entries, and exclude processes:
(Fill in USERID)
(EXCLUDE_REGEX=A regular expression of process names to exclude. Ex: cron|dovecot)
ausearch -m ALL | perl -0777 -e 'print grep(/uid=USERID/, grep(!/REGEX/im, split(/^----$/m, <>)))'Using user searches (-ua -ue -ui -ul) for ausearch may work too, but I've found it unreliable.
36
« on: March 01, 2017, 09:08:12 pm »
It took me days to get a Windows 7 install back up when I lost a drive with the MBR record that booted to my GPT drive. The windows booting and install processes are just REALLY finicky and temperamental. One of my largest problems was that I couldn’t find certain required files online, so the only way to acquire them was to unhook all but 1 GPT partitioned drive from the computer and install Windows to it. Here are the files needed to boot Windows 7 x64 from a GPT drive, assuming your mother board supports EFI. The first step is creating a system partition anywhere on the drive (you may have to shrink another partition) and extract these files to that partition. This blog post has good instructions on the entire process, however, instead of using bcdboot, I recommend using “bootrec /ScanOS” followed by “bootrec /RebuildBCD”. You MAY also need a “bootrec /FixMBR”. These files were obtained from a Windows 7 x64 Ultimate install, so it should work if your install type matches. I expect it will work for any Windows version of an x64 install. Here is a list of the files:
EFI
├── Boot
│ └── bootx64.efi
└── Microsoft
└── Boot
├── bootmgfw.efi
├── bootmgr.efi
├── BOOTSTAT.DAT
├── cs-CZ
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── da-DK
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── de-DE
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── el-GR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── en-US
│ ├── bootmgfw.efi.mui
│ ├── bootmgr.efi.mui
│ └── memtest.efi.mui
├── es-ES
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── fi-FI
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── Fonts
│ ├── chs_boot.ttf
│ ├── cht_boot.ttf
│ ├── jpn_boot.ttf
│ ├── kor_boot.ttf
│ └── wgl4_boot.ttf
├── fr-FR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── hu-HU
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── it-IT
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── ja-JP
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── ko-KR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── memtest.efi
├── nb-NO
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── nl-NL
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── pl-PL
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── pt-BR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── pt-PT
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── ru-RU
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── sv-SE
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── tr-TR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── zh-CN
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── zh-HK
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
└── zh-TW
├── bootmgfw.efi.mui
└── bootmgr.efi.mui
27 directories, 57 files
“EFI\Microsoft\Boot\BCD” is not included because it is computer dependent and is created with the bootrec command. “EFI\Microsoft\Boot\BCD.LOG*” are not included for obvious reasons.
37
« on: February 18, 2017, 12:38:34 pm »
I decided to try out the PHP IDE PHPStorm by jet brains (you know, the people that made android studio) again recently but ran into a few problems. I tried it a while back and I abandoned it because I was using the browser connect method instead of xdebug and it was a major PITA. This time around, I tried xdebug and everything works much smoother and easier. However, there is a major bug with it in which single line statements inside control statements could not be stopped on in the debugger (breakpoint or stepping). This has been a problem since 2012 and was marked until very recently as unfixable. I am honestly shocked this is not a thing more cited. Do people just not use that form of formatting anymore? I find it much cleaner. Example:
if(true)
$a=1; //This line could never be reached
So all code like this had to be changed to the following for debugging.
if(true) {
$a=1;
}
However, in a comment by one of the developers on 2016-12-11 (just 2 months ago) he said “This turned out to be a duplicate of 1165, for which I now have cooked up a fix, which will make it into 2.5.1.”. Unfortunately, there has not yet been a release of this version, so I was forced to a compile a copy myself on Windows. This turned out to also be a major PITA. I have documented the process here. Here is the version I compiled (7.1.2 threadsafe x86) - To compile a php extension, you also have to compile php itself. So first download both the source and zip files for your chosen version. I used the latest version, 7.1.2 VC14 x86 Thread Safe (2017-Feb-14 23:28:41) [7.1.2 source]. The source is needed for obvious reasons. The binary is needed so we can extract some dlls not included with the source. Extract the php source root directory files to “c:\php\src.”
- You also need to grab the xdebug source [github] via “git clone git://github.com/xdebug/xdebug.git”. I used commit #62ac284bf36f7521e78d2b21492ce2899f7cc4ff
#379ab7ab68d28d0af8f637aa28c7e9fab921f27a, but it actually had a bug in it which I fixed in my fork. I sent a pull request so it should hopefully be integrated soon. Clone the git repo in “c:\php\src-extensions” (it will create the “xdebug” directory inside of it) - You will also need visual studio 14.0 (VS 2015) [direct download] installed.
- Once visual studio is installed, you need to open “Developer Command Prompt for VS2015”. It can be found in your start menu under “Programs\Visual Studio 2015\Visual Studio Tools”.
- Run the following commands to get php compiled:
cd\php\src
buildconf --add-modules-dir=..\src-extensions
:: You may want some different flags for the below command
configure "--enable-snapshot-build" "--enable-debug-pack" "--without-pdo-mssql" "--enable-com-dotnet=shared" "--with-mcrypt=static" "--without-analyzer"
:: At this point we are going to actually build php to get the phpize.bat
:: You may receive some warnings at the end that say something to the extent of
:: “'-9' is not recognized as an internal or external command”. You can ignore these
nmake snap
:: This next command will “install” php to “c:\php”.
nmake install
- Next we get to compile xdebug. But it requirements bison, which can be downloaded here [Direct download]. Extract bin/bison.exe and bin/m4.exe to the VC/bin directory, most likely at “C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin”.
For some reason, the first time I did this process I needed the following files also in VC/bin directory, all found at http://gnuwin32.sourceforge.net. However, during my second run to create this tutorial, I did not need them. iconv.exe, libcharset1.dll, libiconv2.dll, libintl3.dll, mspdb110.dll, regex2.dll. And the “share” folder from the bison zip file needed to be extracted to “C:\Program Files (x86)”. - Some file edits in the xdebug directory:
- Add the following line to the top of configure.js: “PHP_SECURITY_FLAGS=true;”
- If you are using the commit I mentioned, Add the following to line 471 of xdebug_code_coverage.c “zend_op *base_address = &(opa->opcodes[0]);”
- Also edit line 9 of template.rc by changing it from “#include "main/php_version.h"” to “#include "../../src/main/php_version.h"”
- And now to actually compile xdebug
cd ..\src-extensions\xdebug
..\..\SDK\phpize
configure --with-xdebug
nmake php_xdebug.dll
copy Release_TS\php_xdebug.dll ..\..\ext
- Your complete php install is now located in “c:\php”. But exclude the “src” and “src-extensions” folder. It still needs a few dlls that are found in the php zip file you downloaded earlier. Extract all the dlls from that root of that file to your root php folder, but do not overwrite anything that is already there. If you wanted to compile your own version of the apache dll, you can add one of “--enable-apache2handler, --enable-apache2-2handler, --enable-apache2-4handler” to the src/configure command.
- For some really odd reason, my version of php would always give me the following error “Packets out of order. Expected 3 received 2. Packet size=22” when trying to connect to mysql (both pdo and mysqli go through mysqlnd). So I just ended up using the php files provided through the zip file, with my newly compiled xdebug dll.
- Definitely have to mention that the following blog post was a big help to me in figuring this all out: http://blog.benoitblanchon.fr/build-php-extension-on-windows/
38
« on: February 17, 2017, 02:47:07 am »
Same disclaimer. Guess I gotta add command line flags for this and update on github. Already have 1 pull request on github. PITA cause I don't release code without thoroughly testing.
Line 137. Change
sys.exit("File not found in DB: "+FilePath)to
continueWhen I update on github it will throw a warning instead.
39
« on: February 16, 2017, 03:56:28 am »
Had a need today to sync file modification timestamps between plex and the actual files. Code is as follows.
#Dump from sqlite (DB should be in C:\Users\USERNAME\AppData\Local\Plex Media Server\Plug-in Support\Databases
sqlite3 com.plexapp.plugins.library.db 'select file, updated_at from media_parts' | \
sort | uniq | perl -pe 's/\\/\//g' | perl -pe 's/^\w:/\./gi' | perl -pe 's/\|/\t/g' | \
grep -viP '^((DRIVES_TO_IGNORE):|\./(DIRECTORIES_TO_IGNORE)|\|$)' > NAME_CHECKS
#Find all files with their modification timestamps
find -L -type f -printf "%p\t%TY-%Tm-%Td %TH:%TM:%.2TS\n" | sort > FTIMES2
#Filter out unwanted folders and file extensions. I did this as a separate command from the above line to allow filtering without having to run a find on the entire drive again
cat FTIMES2 | grep -vP '\.(md5|torrent|sub|idx|nfo|srt|txt|ssa|log|db|jpg|tbn|sfv|png|cbz|rar|cbr|OTHER_EXTENSIONS)\t' | grep -vP '^./(System Volume Information|\$RECYCLE\.BIN|OTHER_FOLDERS)/' > FTIMES
#After comparing the 2 files and extracting any files that need to be updated, run this regular expression on the data to get touch commands to update the timestamps
^(.*)\t\d\d(\d\d)-(\d\d)-(\d\d) (\d\d):(\d\d):(\d\d)$ => touch -m -t $2$3$4$5$6.$7 "$1"
40
« on: February 14, 2017, 10:23:31 am »
In PlexPlaylistImporter.py, below line 177 #Insert the items into the playlistAdd the following line
Cur.execute('DELETE FROM play_queue_generators WHERE playlist_id=?', (PlexPlaylistID,))
I have not tested this code, so make sure to backup your database before attempting it.
41
« on: January 20, 2017, 09:05:24 am »
List joined hard links. Same links are separated by tab with a newline between each set.
find -links +1 | xargs -d"\n" ls -i | perl -e 'foreach $Line (<STDIN>) { @Cols=($Line=~/^\s*(\d+)\s*(.*?)\s*$/); push(@{$Link{$Cols[0]}}, $Cols[1]); } foreach $List (values %Link) { print join("\t", @{$List})."\n"; }'
42
« on: January 06, 2017, 04:15:20 am »
Just something that might be useful to others. I couldn’t find a copy of this anywhere else online. This should contain every point on the map. If anyone finds any more, please let me know! 
43
« on: November 10, 2016, 04:52:15 pm »
Regular expressions have been one of my favorite programming tools since I first discovered them. They are wonderfully robust and things can usually be done with them in many ways. For example, here are multiple ways to match an IPv4 address: - ^\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?$
- ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$
- ^(\d{1,3}\.){3}\d{1,3}$
- ^([0-9]{1,3}\.){3}[0-9]{1,3}$
One of my major annoyances though has always been lists. I have always done them like ^(REGEX,)*REGEX$. For example, I would do a list of IP addresses like this: ^(\d{1,3}\.){3}\d{1,3},)*\d{1,3}\.){3}\d{1,3}$. I recently realized however that a list can much more elegantly be done as follows: ^(REGEX(,|$))+(?<!,)$. I would describe this as working by: - ^: Start of the statement (test string)
- (REGEX(,|$))+: A list of items separated by either a comma or EOS (end of statement). If we keep this regular expression as not-multi-line (the default), then the EOS can only happen at the end of the statement.
- (?<!,): This is a look-behind assertion saying that the last character before the EOS cannot be a comma. If we didn’t have this, the list could look like this, with a comma at the end: “ITEM,ITEM,ITEM,”.
- $: The end of the statement
So the new version of the IP address list would look like this ^((\d{1,3}\.){3}\d{1,3}(,|$))+(?<!,)$ instead of this ^((\d{1,3}\.){3}\d{1,3},)*(\d{1,3}\.){3}\d{1,3}$. Also, since an IP address is just a list of numbers separated by periods, it could also look like this: ^(\d{1,3}(\.|$)){4}(?<!\.)$.
44
« on: October 27, 2016, 02:46:15 am »
I wanted to write about a really weird problem I recently had while debugging in C++ (technically, it’s all C). Unfortunately, I was doing this in kernel debugging mode, which made life a bit harder, but it would have happened the same in userland. I had an .hpp file (we’ll call it process_internal.hpp) that was originally an internal file just to be included from a .cpp file (we’ll call it process.cpp), so it contained global variables as symbols. I ended up needing to include this process_internal.hpp file elsewhere (for testing, we’ll call it test.cpp). Because of this, the same symbol was included in multiple files, so the separate .o builds were not properly interacting. I ended up using “#ifdef”s to only include the parts I needed in the test.cpp file, and doing “extern” defines of the global variables for it. It looked something like the following:
enum { FT_Inbound, FT_Outbound };
typedef struct FilteringLayer {
int FilterTypeNum, OriginalID;
const char *Name;
} FilteringLayer;
const int FT_NumTypes=2;
#ifdef _PROCESS_INTERNAL
FilteringLayer FilterTypes[FT_NumTypes]={
{FT_Inbound, 5, "Inbound"),
{FT_Outbound, 8, "Outbound"),
};
#else
extern "C" FilteringLayer *FilterTypes;
#endif
So I was accessing this variable in test.cpp and getting a really weird problem. The code looked something like this:
struct foo { int a, b; };
foo Stuff[]={...};
void FunctionBar()
{
for(int i=0;i<FT_NumTypes;i++)
Stuff[FilterTypes[i].OriginalID].b=1;
}
This was causing an access exception, which blue screened my debug VM. I tried running the exact same statements in the visual studio debugger, and things were working just as they were supposed to! So I decided to go to the assembly level. It looked something like this: (I included descriptions) | L# | Code | | | | 1 | mov qword ptr [rsp+58h],0 | | 2 | jmp MODULENAME!FunctionBar+0xef | | 3 | mov rax,qword ptr [rsp+58h] | | 4 | inc rax | | 5 | mov qword ptr [rsp+58h],rax | | 6 | cmp qword ptr [rsp+58h],02h | | 7 | jae MODULENAME!FunctionBar+0x11e | | | | 8 | imul rax,qword ptr [rsp+58h],10h | | 9 | mov rcx,[MODULENAME!FilterTypes ] | | 10 | movzx eax,word ptr [rcx+rax+4] | | 11 | imul rax,rax,30h | | 12 | lea rcx,[MODULENAME!Stuff ] | | 13 | mov dword ptr [rcx+rax+04h],1 | | 14 | jmp MODULENAME!FunctionBar+0xe2 | | 15 |
I noticed that line #9 was putting 0x0000000C`00000000 into RCX instead of &FilterTypes. I knew the instruction should have been an “lea” instead of a “mov” to fix this. My first thought was compiler bug, but as many programming mantras say, that is very very rarely the case. If you want to guess now what the problem is, now is the time. I’ve given you all the information (and more) to make the guess. The answer: extern "C" FilteringLayer *FilterTypes; should have been extern "C" FilteringLayer FilterTypes[];. Oops! The debugger was getting it right because it had the extra information of the real definition of the FilterTypes variable.
45
« on: October 03, 2016, 09:43:08 am »
There are 3 different methods for updating multiple rows at once in MySQL with different values: - INSERT: INSERT with ON DUPLICATE KEY UPDATE
INSERT INTO FooBar (ID, foo)
VALUES (1, 5), (2,  , (3, 2)
ON DUPLICATE KEY UPDATE foo=VALUES(foo);
- TRANSACTION: Where you do an update for each record within a transaction (InnoDB or other DBs with transactions)
START TRANSACTION;
UPDATE FooBar SET foo=5 WHERE ID=1;
UPDATE FooBar SET foo=8 WHERE ID=2;
UPDATE FooBar SET foo=2 WHERE ID=3;
COMMIT;
- CASE: In which you a case/when for each different record within an UPDATE
UPDATE FooBar SET foo=CASE ID
WHEN 1 THEN 5
WHEN 2 THEN 8
WHEN 3 THEN 2
END
WHERE ID IN (1,2,3);
I feel knowing the speeds of the 3 different methods is important. All of the following numbers apply to InnoDB. I just tested this, and the INSERT method was 6.7x faster for me than the TRANSACTION method. I tried on a set of both 3,000 and 30,000 rows and got the same results. The TRANSACTION method still has to run each individually query, which takes time, though it batches the results in memory, or something, while executing. The TRANSACTION method is also pretty expensive in both replication and query logs. Even worse, the CASE method was 41.1x slower than the INSERT method w/ 30,000 records (6.1x slower than TRANSACTION). And 75x slower in MyISAM. INSERT and CASE methods broke even at ~1,000 records. Even at 100 records, the CASE method is BARELY faster. So in general, I feel the INSERT method is both best and easiest to use. The queries are smaller and easier to read and only take up 1 query of action. This applies to both InnoDB and MyISAM. Bonus stuff:Using the INSERT method, there can be a problem in which NON-NULL fields with no default (in other words, required fields) are not being updated. You will get an error like “Field 'fieldname' doesn't have a default value”. The solution is to temporarily turn off STRICT_TRANS_TABLES and STRICT_ALL_TABLES in the SQL mode: SET SESSION sql_mode=REPLACE(REPLACE(@@SESSION.sql_mode,"STRICT_TRANS_TABLES",""),"STRICT_ALL_TABLES",""). Make sure to save the sql_mode first if you plan on reverting it. As for other comments I’ve seen that say the auto_increment goes up using the INSERT method, I tested that too and it seems to not be the case. Code to run the tests is as follows: (It also outputs .SQL files to remove PHP interpreter overhead)
<?
//Variables
$NumRows=30000;
//These 2 functions need to be filled in
function InitSQL()
{
}
function RunSQLQuery($Q)
{
}
//Run the 3 tests
InitSQL();
for($i=0;$i<3;$i++)
RunTest($i, $NumRows);
function RunTest($TestNum, $NumRows)
{
$TheQueries=Array();
$DoQuery=function($Query) use (&$TheQueries)
{
RunSQLQuery($Query);
$TheQueries[]=$Query;
};
$TableName='Test';
$DoQuery('DROP TABLE IF EXISTS '.$TableName);
$DoQuery('CREATE TABLE '.$TableName.' (i1 int NOT NULL AUTO_INCREMENT, i2 int NOT NULL, primary key (i1)) ENGINE=InnoDB');
$DoQuery('INSERT INTO '.$TableName.' (i2) VALUES ('.implode('), (', range(2, $NumRows+1)).')');
if($TestNum==0)
{
$TestName='Transaction';
$Start=microtime(true);
$DoQuery('START TRANSACTION');
for($i=1;$i<=$NumRows;$i++)
$DoQuery('UPDATE '.$TableName.' SET i2='.(($i+5)*1000).' WHERE i1='.$i);
$DoQuery('COMMIT');
}
if($TestNum==1)
{
$TestName='Insert';
$Query=Array();
for($i=1;$i<=$NumRows;$i++)
$Query[]=sprintf("(%d,%d)", $i, (($i+5)*1000));
$Start=microtime(true);
$DoQuery('INSERT INTO '.$TableName.' VALUES '.implode(', ', $Query).' ON DUPLICATE KEY UPDATE i2=VALUES(i2)');
}
if($TestNum==2)
{
$TestName='Case';
$Query=Array();
for($i=1;$i<=$NumRows;$i++)
$Query[]=sprintf('WHEN %d THEN %d', $i, (($i+5)*1000));
$Start=microtime(true);
$DoQuery("UPDATE $TableName SET i2=CASE i1\n".implode("\n", $Query)."\nEND\nWHERE i1 IN (".implode(',', range(1, $NumRows)).')');
}
print "$TestName: ".(microtime(true)-$Start)."
\n";
file_put_contents("./$TestName.sql", implode(";\n", $TheQueries).';');
}
Pages: 1 2 [3] 4 5 ... 37
|
|
