|
-
April 26, 2024, 10:25:52 pm
- Welcome, Guest
News:Official site launch very soon, hurrah!
Show Posts
This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.
Topics - Dakusan
16
« on: December 29, 2018, 04:48:03 am »
#This script takes a newline delimited file list from STDIN for md5 hashing
#This script requires the `md5sum`, `pv`, `paste`, `bc`, and 'numfmt' commands
#The output of the md5s are stored in the file specified by the first parameter
#The format for each md5 hash to the output file is "$FileName\t$Hash\n"
#File sizes are always output in megabytes with 3 decimal places
#While calculating the hashes the script keeps the user informed of the progress of both the current file and all the files as follows:
#1) "Hashing: $FileName ($FileSize MiB)\n"
#2) The progress of the hash of the file ran through `pv`, with the size precalculated for file progress percents
#3) "Finished $TotalProgressPercent% ($ProcessedBytes/$TotalBytes MiB)\n\n"
#Get $Outfile from the first argument and the $FileList from STDIN (newline delimited)
OutFile="$1";
FileList=`cat /dev/stdin`
#Format a byte count in MegaBytes with comma grouping and 3 decimal places
MbFmtNoExt ()
{
echo "scale=3; $1/1024/1024" | bc | echo -n `xargs numfmt --grouping`
}
#Add " MiB" to the end of MbFmtNoExt
MbFmt ()
{
echo `MbFmtNoExt $1`" MiB"
}
#Calculate and output the total size of the file list
echo -n "Calculating total size: "
TotalSize=`echo "$FileList" | xargs -d"\n" stat --printf="%s\n" | paste -s -d+ | bc`
MbFmt $TotalSize
echo #Add an extra newline
#Run over the list (newline delimited)
CalculatedBytes=0
IFS=$'\n'
for FileName in `echo "$FileList"`
do
#Output the file size and name to STDOUT
FileSize=`stat --printf="%s" "$FileName"`
echo "Hashing: $FileName ("`MbFmt $FileSize`")"
#Output the filename to $OutFile
echo -n $FileName$'\t' >> $OutFile
#Run the md5 calculation with `pv` progress
#Output the hash to $OutFile after the FileName and a tab
cat "$FileName" | pv -s $FileSize | md5sum | awk '{print $1}' >> $OutFile
#Output the current progress for the entire file list
#Format: "Finished $TotalProgressPercent% ($ProcessedBytes/$TotalBytes MiB)\n\n"
CalculatedBytes=$(($CalculatedBytes+$FileSize))
echo -n "Finished "
printf "%.3f" `echo "scale=4; $CalculatedBytes*100/$TotalSize" | bc`
echo "% ("`MbFmtNoExt $CalculatedBytes`"/"`MbFmt $TotalSize`$')\n'
done
17
« on: December 07, 2018, 03:29:27 pm »
On my primary computer (whose harddrive is encrypted) I always have Windows auto logging in to help with the bootup time. However, my bootup time can be rather slow; and if I needed to have my computer booted but locked, I had to wait for the login to complete so I could lock the computer. This has been becoming a nuisance lately when I need to get out of my house quickly in the morning. For the solution I created a windows boot entry that auto locks the computer after logging the user in. This also requires a batch file, to run for the user on startup, to detect when this boot entry was selected. Here are the steps to create this setup: - Create the new boot entry:In the windows command line, run: bcdedit /copy {current} /d "Lock on Startup"
This creates a new boot option, duplicated from your currently selected boot option, in the boot menu labeled “Lock on Startup”. - (Optional) Change the bootup timeout:In the windows command line, run: bcdedit /timeout 5
Where 5 is a 5 second timeout. - Create a batch file to run on login:In your start menu’s startup folder, add a batch file. You can name it anything as long as the extension is “.bat”.
Add the following to the file: bcdedit /enum {current} | findstr /r /c:"description *Lock on Startup" && rundll32.exe user32.dll,LockWorkStation
Note that there are 2 spaces in the description search string to replicate the regular expression's 1 or more quantifier “+”, since findstr only supports the 0 or more quantifier “*”.
18
« on: August 30, 2018, 01:56:04 am »
I wanted a simple setup in Symfony where the programmer could define their ide in the parameters file. Sounds simple, right? Just add something like ide_url: 'phpstorm' to parameters.yml->parameters and ide: '%ide_url%' to config.yml->framework. And it worked great, however, my problem was much more convoluted. I am actually running the Symfony server on another machine and am accessing the files via NFS on Windows. So, it would try to open PHPStorm with the incorrect path. Symfony suggests the solution to this is writing your own custom URL handler with %f and %l to fill in the filename and line, and use some weird formatting to do string replaces. So I wrote in 'idea://%%f:%%l&/PROJECT_PATH_ON_SERVER/>DRIVE_LETTER:/PATH_ON_WINDOWS/' (note the double parenthesis for escaping) directly in the config.yml and that worked, kind of. The URL was perfect, but IntelliJ does not seem to register the idea:// protocol handler like PHPStorm theoretically does (according to some online threads) with phpstorm://. So I had to write my own solution. This answer on stackoverflow has the answer on how to register a protocol handler in Windows. But the problem now was that the first parameter passed to IntelliJ started with the idea:// which broke the command-line file-open. So I ended up writing a script to fix this, which is at the bottom. OK, so we’re almost there; I just had to paste the string I came up with back into the parameters.yml, right? I wish. While this was now working properly in a Symfony error page, a new problem arose. The Symfony bin/console debug:config framework command was failing with You have requested a non-existent parameter "f:". The darn thing was reading the unescaped string as 'idea://%f:%l&...' and it thought %f:% was supposed to be a variable. Sigh. So the final part was to double escape the strings with 4 percent signs. 'idea://%%%%f:%%%%l&...'. Except now the URL on the error pages gave me idea://%THE_PATH:%THE_LINE_NUMBER. It was adding an extra parenthesis before both values. This was simple to resolve in the script I wrote, so I was finally able to open scripts directly from the error page. Yay. So here is the final set of data that has to be added to make this work: Registry: HKCR/idea/(default) = URL:idea ProtocolHKCR/idea/URL Protocol = ""HKCR/idea/shell/open/command = "PATH_TO_PHP" -f "PATH_TO_SCRIPT" "%1" "%2" "%3" "%4" "%5" "%6" "%7" "%8" "%9"parameters.yml:parameters:ide_url: 'idea://%%%%f:%%%%l&/ PROJECT_PATH_ON_SERVER/> DRIVE_LETTER:/ PATH_ON_WINDOWS/'config.yml:framework:ide: '%ide_url%'PHP_SCRIPT_FILE:
<?php
function DoOutput($S)
{
//You might want to do something like output the error to a file or do an alert here
print $S;
}
if(!isset($argv[1]))
return DoOutput('File not given');
if(!preg_match('~^idea://(?:%25|%)?([a-z]:[/\\\\][^:]+):%?(\d+)/?$~i', $argv[1], $MatchData))
return DoOutput('Invalid format: '.$argv[1]);
$FilePath=$MatchData[1];
if(!file_exists($FilePath))
return DoOutput('Cannot find file: '.$FilePath);
$String='"C:\Program Files\JetBrains\IntelliJ IDEA 2018.1.6\bin\idea64.exe" --line '.$MatchData[2].' '.escapeshellarg($FilePath);
DoOutput($String);
shell_exec($String);
?>
19
« on: August 01, 2018, 10:01:31 pm »
I was surprised in my failure to find a script online to download all of an author’s stories from Fiction Press or Fan Fiction.Net, so I threw together the below. If you go to an author’s page in a browser (only tested in Chrome) it should have all of their stories, and you can run the following script in the console (F12) to grab them all. Their save name format is STORY_NAME_LINK_FORMAT - CHAPTER_NUMBER.html. It works as follows: - Gathers all of the names, chapter 1 links, and chapter counts for each story.
- Converts this information into a list of links it needs to download. The links are formed by using the chapter 1 link, and just replacing the chapter number.
- It then downloads all of the links to your current browser’s download folder.
Do note that chrome should prompt you to answer “This site is attempting to download multiple files”. So of course, say yes. The script is also designed to detect problems, which would happen if fictionpress changes their html formatting.
//Gather the story information
const Stories=[];
$('.mystories .stitle').each((Index, El) =>
Stories[Index]={Link:$(El).attr('href'), Name:$(El).text()}
);
$('.mystories .xgray').each((Index, El) =>
Stories[Index].NumChapters=/ - Chapters: (\d+) - /.exec($(El).text())[1]
);
//Get links to all stories
const LinkStart=document.location.protocol+'//'+document.location.host;
const AllLinks=[];
$.each(Stories, (_, Story) => {
if(typeof(Story.NumChapters)!=='string' || !/^\d+$/.test(Story.NumChapters))
return console.log('Bad number of chapters for: '+Story.Name);
const StoryParts=/^\/s\/(\d+)\/1\/(.*)$/.exec(Story.Link);
if(!StoryParts)
return console.log('Bad link format for stories: '+Story.Name);
for(let i=1; i<=Story.NumChapters; i++)
AllLinks.push([LinkStart+'/s/'+StoryParts[1]+'/'+i+'/'+StoryParts[2], StoryParts[2]+' - '+i+'.html']);
});
//Download all the links
$.each(AllLinks, (_, LinkInfo) =>
$('a').attr('download', LinkInfo[1]).attr('href', LinkInfo[0])[0].click()
);
jQuery('.blurb.group .heading a[href^="/works"]').map((_, El) => jQuery(El).text()).toArray().join('\n');
20
« on: October 10, 2017, 06:35:19 pm »
The following is a simple bash script to ping a different domain once a second and log the output. By default, it pings #.castledragmire.com, where # is an incrementing number starting from 0. The script is written for Cygwin (See the PING_COMMAND variable at the top) but is very easily adaptable to Linux. The log output is: EPOCH_TIMESTAMP DOMAIN PING_OUTPUT
#This uses Window's native ping since the Cygwin ping is sorely lacking in options
#"-n 1"=Only runs once, "-w 3000"=Timeout after 3 seconds
#The grep strings are also directly tailored for Window's native ping
PING_COMMAND=$(
echo 'C:/Windows/System32/PING.EXE -n 1 -w 3000 $DOMAIN |';
echo 'grep -iP "^(Request timed out|Reply from|Ping request could not find)"';
)
i=0 #The subdomain counter
STARTTIME=`date +%s.%N` #This holds the timestamp of the end of the previous loop
#Infinite loop
while true
do
#Get the domain to run. This requires a domain that has a wildcard as a primary subdomain
DOMAIN="$i.castledragmire.com"
#Output the time, domain name, and ping output
echo `date +%s` "$DOMAIN" $(eval $PING_COMMAND)
#If less than a second has passed, sleep up to 1 second
ENDTIME=`date +%s.%N`
SLEEPTIME=$(echo "1 - ($ENDTIME - $STARTTIME)" | bc)
STARTTIME=$ENDTIME
if [ $(echo "$SLEEPTIME>0" | bc) -eq 1 ]; then
sleep $SLEEPTIME
STARTTIME=$(echo "$STARTTIME + $SLEEPTIME" | bc)
fi
#Increment the subdomain counter
let i+=1
done
21
« on: March 01, 2017, 09:08:12 pm »
It took me days to get a Windows 7 install back up when I lost a drive with the MBR record that booted to my GPT drive. The windows booting and install processes are just REALLY finicky and temperamental. One of my largest problems was that I couldn’t find certain required files online, so the only way to acquire them was to unhook all but 1 GPT partitioned drive from the computer and install Windows to it. Here are the files needed to boot Windows 7 x64 from a GPT drive, assuming your mother board supports EFI. The first step is creating a system partition anywhere on the drive (you may have to shrink another partition) and extract these files to that partition. This blog post has good instructions on the entire process, however, instead of using bcdboot, I recommend using “bootrec /ScanOS” followed by “bootrec /RebuildBCD”. You MAY also need a “bootrec /FixMBR”. These files were obtained from a Windows 7 x64 Ultimate install, so it should work if your install type matches. I expect it will work for any Windows version of an x64 install. Here is a list of the files:
EFI
├── Boot
│ └── bootx64.efi
└── Microsoft
└── Boot
├── bootmgfw.efi
├── bootmgr.efi
├── BOOTSTAT.DAT
├── cs-CZ
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── da-DK
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── de-DE
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── el-GR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── en-US
│ ├── bootmgfw.efi.mui
│ ├── bootmgr.efi.mui
│ └── memtest.efi.mui
├── es-ES
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── fi-FI
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── Fonts
│ ├── chs_boot.ttf
│ ├── cht_boot.ttf
│ ├── jpn_boot.ttf
│ ├── kor_boot.ttf
│ └── wgl4_boot.ttf
├── fr-FR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── hu-HU
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── it-IT
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── ja-JP
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── ko-KR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── memtest.efi
├── nb-NO
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── nl-NL
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── pl-PL
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── pt-BR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── pt-PT
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── ru-RU
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── sv-SE
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── tr-TR
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── zh-CN
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
├── zh-HK
│ ├── bootmgfw.efi.mui
│ └── bootmgr.efi.mui
└── zh-TW
├── bootmgfw.efi.mui
└── bootmgr.efi.mui
27 directories, 57 files
“EFI\Microsoft\Boot\BCD” is not included because it is computer dependent and is created with the bootrec command. “EFI\Microsoft\Boot\BCD.LOG*” are not included for obvious reasons.
22
« on: February 18, 2017, 12:38:34 pm »
I decided to try out the PHP IDE PHPStorm by jet brains (you know, the people that made android studio) again recently but ran into a few problems. I tried it a while back and I abandoned it because I was using the browser connect method instead of xdebug and it was a major PITA. This time around, I tried xdebug and everything works much smoother and easier. However, there is a major bug with it in which single line statements inside control statements could not be stopped on in the debugger (breakpoint or stepping). This has been a problem since 2012 and was marked until very recently as unfixable. I am honestly shocked this is not a thing more cited. Do people just not use that form of formatting anymore? I find it much cleaner. Example:
if(true)
$a=1; //This line could never be reached
So all code like this had to be changed to the following for debugging.
if(true) {
$a=1;
}
However, in a comment by one of the developers on 2016-12-11 (just 2 months ago) he said “This turned out to be a duplicate of 1165, for which I now have cooked up a fix, which will make it into 2.5.1.”. Unfortunately, there has not yet been a release of this version, so I was forced to a compile a copy myself on Windows. This turned out to also be a major PITA. I have documented the process here. Here is the version I compiled (7.1.2 threadsafe x86) - To compile a php extension, you also have to compile php itself. So first download both the source and zip files for your chosen version. I used the latest version, 7.1.2 VC14 x86 Thread Safe (2017-Feb-14 23:28:41) [7.1.2 source]. The source is needed for obvious reasons. The binary is needed so we can extract some dlls not included with the source. Extract the php source root directory files to “c:\php\src.”
- You also need to grab the xdebug source [github] via “git clone git://github.com/xdebug/xdebug.git”. I used commit #62ac284bf36f7521e78d2b21492ce2899f7cc4ff
#379ab7ab68d28d0af8f637aa28c7e9fab921f27a, but it actually had a bug in it which I fixed in my fork. I sent a pull request so it should hopefully be integrated soon. Clone the git repo in “c:\php\src-extensions” (it will create the “xdebug” directory inside of it) - You will also need visual studio 14.0 (VS 2015) [direct download] installed.
- Once visual studio is installed, you need to open “Developer Command Prompt for VS2015”. It can be found in your start menu under “Programs\Visual Studio 2015\Visual Studio Tools”.
- Run the following commands to get php compiled:
cd\php\src
buildconf --add-modules-dir=..\src-extensions
:: You may want some different flags for the below command
configure "--enable-snapshot-build" "--enable-debug-pack" "--without-pdo-mssql" "--enable-com-dotnet=shared" "--with-mcrypt=static" "--without-analyzer"
:: At this point we are going to actually build php to get the phpize.bat
:: You may receive some warnings at the end that say something to the extent of
:: “'-9' is not recognized as an internal or external command”. You can ignore these
nmake snap
:: This next command will “install” php to “c:\php”.
nmake install
- Next we get to compile xdebug. But it requirements bison, which can be downloaded here [Direct download]. Extract bin/bison.exe and bin/m4.exe to the VC/bin directory, most likely at “C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin”.
For some reason, the first time I did this process I needed the following files also in VC/bin directory, all found at http://gnuwin32.sourceforge.net. However, during my second run to create this tutorial, I did not need them. iconv.exe, libcharset1.dll, libiconv2.dll, libintl3.dll, mspdb110.dll, regex2.dll. And the “share” folder from the bison zip file needed to be extracted to “C:\Program Files (x86)”. - Some file edits in the xdebug directory:
- Add the following line to the top of configure.js: “PHP_SECURITY_FLAGS=true;”
- If you are using the commit I mentioned, Add the following to line 471 of xdebug_code_coverage.c “zend_op *base_address = &(opa->opcodes[0]);”
- Also edit line 9 of template.rc by changing it from “#include "main/php_version.h"” to “#include "../../src/main/php_version.h"”
- And now to actually compile xdebug
cd ..\src-extensions\xdebug
..\..\SDK\phpize
configure --with-xdebug
nmake php_xdebug.dll
copy Release_TS\php_xdebug.dll ..\..\ext
- Your complete php install is now located in “c:\php”. But exclude the “src” and “src-extensions” folder. It still needs a few dlls that are found in the php zip file you downloaded earlier. Extract all the dlls from that root of that file to your root php folder, but do not overwrite anything that is already there. If you wanted to compile your own version of the apache dll, you can add one of “--enable-apache2handler, --enable-apache2-2handler, --enable-apache2-4handler” to the src/configure command.
- For some really odd reason, my version of php would always give me the following error “Packets out of order. Expected 3 received 2. Packet size=22” when trying to connect to mysql (both pdo and mysqli go through mysqlnd). So I just ended up using the php files provided through the zip file, with my newly compiled xdebug dll.
- Definitely have to mention that the following blog post was a big help to me in figuring this all out: http://blog.benoitblanchon.fr/build-php-extension-on-windows/
23
« on: January 06, 2017, 04:15:20 am »
Just something that might be useful to others. I couldn’t find a copy of this anywhere else online. This should contain every point on the map. If anyone finds any more, please let me know! 
24
« on: November 10, 2016, 04:52:15 pm »
Regular expressions have been one of my favorite programming tools since I first discovered them. They are wonderfully robust and things can usually be done with them in many ways. For example, here are multiple ways to match an IPv4 address: - ^\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?$
- ^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$
- ^(\d{1,3}\.){3}\d{1,3}$
- ^([0-9]{1,3}\.){3}[0-9]{1,3}$
One of my major annoyances though has always been lists. I have always done them like ^(REGEX,)*REGEX$. For example, I would do a list of IP addresses like this: ^(\d{1,3}\.){3}\d{1,3},)*\d{1,3}\.){3}\d{1,3}$. I recently realized however that a list can much more elegantly be done as follows: ^(REGEX(,|$))+(?<!,)$. I would describe this as working by: - ^: Start of the statement (test string)
- (REGEX(,|$))+: A list of items separated by either a comma or EOS (end of statement). If we keep this regular expression as not-multi-line (the default), then the EOS can only happen at the end of the statement.
- (?<!,): This is a look-behind assertion saying that the last character before the EOS cannot be a comma. If we didn’t have this, the list could look like this, with a comma at the end: “ITEM,ITEM,ITEM,”.
- $: The end of the statement
So the new version of the IP address list would look like this ^((\d{1,3}\.){3}\d{1,3}(,|$))+(?<!,)$ instead of this ^((\d{1,3}\.){3}\d{1,3},)*(\d{1,3}\.){3}\d{1,3}$. Also, since an IP address is just a list of numbers separated by periods, it could also look like this: ^(\d{1,3}(\.|$)){4}(?<!\.)$.
25
« on: October 27, 2016, 02:46:15 am »
I wanted to write about a really weird problem I recently had while debugging in C++ (technically, it’s all C). Unfortunately, I was doing this in kernel debugging mode, which made life a bit harder, but it would have happened the same in userland. I had an .hpp file (we’ll call it process_internal.hpp) that was originally an internal file just to be included from a .cpp file (we’ll call it process.cpp), so it contained global variables as symbols. I ended up needing to include this process_internal.hpp file elsewhere (for testing, we’ll call it test.cpp). Because of this, the same symbol was included in multiple files, so the separate .o builds were not properly interacting. I ended up using “#ifdef”s to only include the parts I needed in the test.cpp file, and doing “extern” defines of the global variables for it. It looked something like the following:
enum { FT_Inbound, FT_Outbound };
typedef struct FilteringLayer {
int FilterTypeNum, OriginalID;
const char *Name;
} FilteringLayer;
const int FT_NumTypes=2;
#ifdef _PROCESS_INTERNAL
FilteringLayer FilterTypes[FT_NumTypes]={
{FT_Inbound, 5, "Inbound"),
{FT_Outbound, 8, "Outbound"),
};
#else
extern "C" FilteringLayer *FilterTypes;
#endif
So I was accessing this variable in test.cpp and getting a really weird problem. The code looked something like this:
struct foo { int a, b; };
foo Stuff[]={...};
void FunctionBar()
{
for(int i=0;i<FT_NumTypes;i++)
Stuff[FilterTypes[i].OriginalID].b=1;
}
This was causing an access exception, which blue screened my debug VM. I tried running the exact same statements in the visual studio debugger, and things were working just as they were supposed to! So I decided to go to the assembly level. It looked something like this: (I included descriptions) | L# | Code | | | | 1 | mov qword ptr [rsp+58h],0 | | 2 | jmp MODULENAME!FunctionBar+0xef | | 3 | mov rax,qword ptr [rsp+58h] | | 4 | inc rax | | 5 | mov qword ptr [rsp+58h],rax | | 6 | cmp qword ptr [rsp+58h],02h | | 7 | jae MODULENAME!FunctionBar+0x11e | | | | 8 | imul rax,qword ptr [rsp+58h],10h | | 9 | mov rcx,[MODULENAME!FilterTypes ] | | 10 | movzx eax,word ptr [rcx+rax+4] | | 11 | imul rax,rax,30h | | 12 | lea rcx,[MODULENAME!Stuff ] | | 13 | mov dword ptr [rcx+rax+04h],1 | | 14 | jmp MODULENAME!FunctionBar+0xe2 | | 15 |
I noticed that line #9 was putting 0x0000000C`00000000 into RCX instead of &FilterTypes. I knew the instruction should have been an “lea” instead of a “mov” to fix this. My first thought was compiler bug, but as many programming mantras say, that is very very rarely the case. If you want to guess now what the problem is, now is the time. I’ve given you all the information (and more) to make the guess. The answer: extern "C" FilteringLayer *FilterTypes; should have been extern "C" FilteringLayer FilterTypes[];. Oops! The debugger was getting it right because it had the extra information of the real definition of the FilterTypes variable.
26
« on: October 03, 2016, 09:43:08 am »
There are 3 different methods for updating multiple rows at once in MySQL with different values: - INSERT: INSERT with ON DUPLICATE KEY UPDATE
INSERT INTO FooBar (ID, foo)
VALUES (1, 5), (2,  , (3, 2)
ON DUPLICATE KEY UPDATE foo=VALUES(foo);
- TRANSACTION: Where you do an update for each record within a transaction (InnoDB or other DBs with transactions)
START TRANSACTION;
UPDATE FooBar SET foo=5 WHERE ID=1;
UPDATE FooBar SET foo=8 WHERE ID=2;
UPDATE FooBar SET foo=2 WHERE ID=3;
COMMIT;
- CASE: In which you a case/when for each different record within an UPDATE
UPDATE FooBar SET foo=CASE ID
WHEN 1 THEN 5
WHEN 2 THEN 8
WHEN 3 THEN 2
END
WHERE ID IN (1,2,3);
I feel knowing the speeds of the 3 different methods is important. All of the following numbers apply to InnoDB. I just tested this, and the INSERT method was 6.7x faster for me than the TRANSACTION method. I tried on a set of both 3,000 and 30,000 rows and got the same results. The TRANSACTION method still has to run each individually query, which takes time, though it batches the results in memory, or something, while executing. The TRANSACTION method is also pretty expensive in both replication and query logs. Even worse, the CASE method was 41.1x slower than the INSERT method w/ 30,000 records (6.1x slower than TRANSACTION). And 75x slower in MyISAM. INSERT and CASE methods broke even at ~1,000 records. Even at 100 records, the CASE method is BARELY faster. So in general, I feel the INSERT method is both best and easiest to use. The queries are smaller and easier to read and only take up 1 query of action. This applies to both InnoDB and MyISAM. Bonus stuff:Using the INSERT method, there can be a problem in which NON-NULL fields with no default (in other words, required fields) are not being updated. You will get an error like “Field 'fieldname' doesn't have a default value”. The solution is to temporarily turn off STRICT_TRANS_TABLES and STRICT_ALL_TABLES in the SQL mode: SET SESSION sql_mode=REPLACE(REPLACE(@@SESSION.sql_mode,"STRICT_TRANS_TABLES",""),"STRICT_ALL_TABLES",""). Make sure to save the sql_mode first if you plan on reverting it. As for other comments I’ve seen that say the auto_increment goes up using the INSERT method, I tested that too and it seems to not be the case. Code to run the tests is as follows: (It also outputs .SQL files to remove PHP interpreter overhead)
<?
//Variables
$NumRows=30000;
//These 2 functions need to be filled in
function InitSQL()
{
}
function RunSQLQuery($Q)
{
}
//Run the 3 tests
InitSQL();
for($i=0;$i<3;$i++)
RunTest($i, $NumRows);
function RunTest($TestNum, $NumRows)
{
$TheQueries=Array();
$DoQuery=function($Query) use (&$TheQueries)
{
RunSQLQuery($Query);
$TheQueries[]=$Query;
};
$TableName='Test';
$DoQuery('DROP TABLE IF EXISTS '.$TableName);
$DoQuery('CREATE TABLE '.$TableName.' (i1 int NOT NULL AUTO_INCREMENT, i2 int NOT NULL, primary key (i1)) ENGINE=InnoDB');
$DoQuery('INSERT INTO '.$TableName.' (i2) VALUES ('.implode('), (', range(2, $NumRows+1)).')');
if($TestNum==0)
{
$TestName='Transaction';
$Start=microtime(true);
$DoQuery('START TRANSACTION');
for($i=1;$i<=$NumRows;$i++)
$DoQuery('UPDATE '.$TableName.' SET i2='.(($i+5)*1000).' WHERE i1='.$i);
$DoQuery('COMMIT');
}
if($TestNum==1)
{
$TestName='Insert';
$Query=Array();
for($i=1;$i<=$NumRows;$i++)
$Query[]=sprintf("(%d,%d)", $i, (($i+5)*1000));
$Start=microtime(true);
$DoQuery('INSERT INTO '.$TableName.' VALUES '.implode(', ', $Query).' ON DUPLICATE KEY UPDATE i2=VALUES(i2)');
}
if($TestNum==2)
{
$TestName='Case';
$Query=Array();
for($i=1;$i<=$NumRows;$i++)
$Query[]=sprintf('WHEN %d THEN %d', $i, (($i+5)*1000));
$Start=microtime(true);
$DoQuery("UPDATE $TableName SET i2=CASE i1\n".implode("\n", $Query)."\nEND\nWHERE i1 IN (".implode(',', range(1, $NumRows)).')');
}
print "$TestName: ".(microtime(true)-$Start)."
\n";
file_put_contents("./$TestName.sql", implode(";\n", $TheQueries).';');
}
27
« on: September 27, 2016, 02:08:36 pm »
- DSQL v2.0.2.2
- Added ability via DSQL->$StrictMode variable to update MySQL strict mode
- Added DSQL->RawQuery() function which takes just a query, and does no modification on it. I found this required due to extremely slow times on building very large queries.
- Added DSQL->EscapeString() function which escapes a string for use in a mysql query. It has an optional parameter that adds single quotes around the result.
- Added DSQL::PrepareInsertList() function which takes an array of names, and turns it into the following string: ['a', 'b', 'c'] => “(`a`, `b`, `c`) VALUES (?, ?, ?)”
- DWCF v1.1.1
- Added previous variable lookup to GetVars.VarArray.SQLLookup via %THEVAR-$VARNAME%
28
« on: September 27, 2016, 06:04:18 am »
I recently had a Linux client that was, for whatever odd reason, making infinite recursive HTTP calls to a single script, which was making the server process count skyrocket. I decided to use the same module as I did in my Painless migration from PHP MySQL to MySQLi post, which is to say, overriding base functions for fun and profit using the PHP runkit extension. I did this so I could gather, for debugging, logs of when and where the calls that were causing this to occur. The below code overrides all functions listed on the line that says “List of functions to intercept” [Line 9]. It works by first renaming these built in functions to “OVERRIDE_$FuncName” [Line 12], and replacing them with a call to “GlobalRunFunc()” [Line 13], which receives the original function name and argument list. The GlobalRunFunc(): - Checks to see if it is interested in logging the call
- In the case of this example, it will log the call if [Line 20]:
- Line 21: curl_setopt is called with the CURLOPT_URL parameter (enum=10002)
- Line 22: curl_init is called with a first parameter, which would be a URL
- Line 23: file_get_contents or fopen is called and is not an absolute path
(Wordpress calls everything absolutely. Normally I would have only checked for http[s] calls).
- If it does want to log the call, it stores it in a global array (which holds all the calls we will want to log).
The logged data includes [Line 25]:- The function name
- The function parameters
- 2 functions back of backtrace (which can often get quite large when stored in the log file)
- It then calls the original function, with parameters intact, and passes through the return [Line 27].
The “GlobalShutdown()” [Line 30] is then called when the script is closing [Line 38] and saves all the logs, if any exist, to “$GlobalLogDir/$DATETIME.srl”. I have it using “serialize()” to encode the log data [Line 25], as opposed to “json_encode()” or “print_r()” calls, as the latter were getting too large for the logs. You may want to have it use one of these other encoding functions for easier log perusal, if running out of space is not a concern.
<?
//The log data to save is stored here
global $GlobalLogArr, $GlobalLogDir;
$GlobalLogArr=Array();
$GlobalLogDir='./LOG_DIRECTORY_NAME';
//Override the functions here to instead have them call to GlobalRunFunc, which will in turn call the original functions
foreach(Array(
'fopen', 'file_get_contents', 'curl_init', 'curl_setopt', //List of functions to intercept
) as $FuncName)
{
runkit_function_rename($FuncName, "OVERRIDE_$FuncName");
runkit_function_add($FuncName, '', "return GlobalRunFunc('$FuncName', func_get_args());");
}
//This optionally
function GlobalRunFunc($FuncName, $Args)
{
global $GlobalLogArr;
if(
($FuncName=='curl_setopt' && $Args[1]==10002) || //CURLOPT enumeration can be found at https://curl.haxx.se/mail/archive-2004-07/0100.html
($FuncName=='curl_init' && isset($Args[0])) ||
(($FuncName=='file_get_contents' || $FuncName=='fopen') && $Args[0][0]!='/')
)
$GlobalLogArr[]=serialize(Array('FuncName'=>$FuncName, 'Args'=>$Args, 'Trace'=>array_slice(debug_backtrace(), 1, 2)));
return call_user_func_array("OVERRIDE_$FuncName", $Args);
}
function GlobalShutdown()
{
global $GlobalLogArr, $GlobalLogDir;
$Time=microtime(true);
if(count($GlobalLogArr))
file_put_contents($GlobalLogDir.date('Y-m-d_H:i:s.'.substr($Time-floor($Time), 2, 3), floor($Time)).'.srl', implode("\n", $GlobalLogArr));
}
register_shutdown_function('GlobalShutdown');
?>
29
« on: September 26, 2016, 12:37:37 am »
I wrote the code at the end of this post to test the different forms of string concatenation and they really are all almost exactly equal in both memory and time footprints. The two primary methods I used are concatenating strings onto each other, and filling an array with strings and then imploding them. I did 500 string additions with a 1MB string in PHP 5.6 (so the result is a 500MB string). At every iteration of the test, all memory and time footprints were very very close (at ~$IterationNumber*1MB). The runtime of both tests was 50.398 seconds and 50.843 seconds consecutively which are most likely within acceptable margins of error. Garbage collection of strings that are no longer referenced seems to be pretty immediate, even without ever leaving the scope. Since the strings are mutable, no extra memory is really required after the fact. HOWEVER, The following tests showed that there is a different in peak memory usage WHILE the strings are being concatenated.
$OneMB=str_repeat('x', 1024*1024);
$Final=$OneMB.$OneMB.$OneMB.$OneMB.$OneMB;
print memory_get_peak_usage();
Result=10,806,800 bytes (~10MB w/o the initial PHP memory footprint)
$OneMB=str_repeat('x', 1024*1024);
$Final=implode('', Array($OneMB, $OneMB, $OneMB, $OneMB, $OneMB));
print memory_get_peak_usage();
Result=6,613,320 bytes (~6MB w/o the initial PHP memory footprint) So there is in fact a difference that could be significant in very very large string concatenations memory-wise (I have run into such examples when creating very large data sets or SQL queries). But even this fact is disputable depending upon the data. For example, concatenating 1 character onto a string to get 50 million bytes (so 50 million iterations) took a maximum amount of 50,322,512 bytes (~48MB) in 5.97 seconds. While doing the array method ended up using 7,337,107,176 bytes (~6.8GB) to create the array in 12.1 seconds, and then took an extra 4.32 seconds to combine the strings from the array. Anywho... the below is the benchmark code I mentioned at the beginning which shows the methods are pretty much equal. It outputs a pretty HTML table.
<?
//Please note, for the recursion test to go beyond 256, xdebug.max_nesting_level needs to be raised.
//You also may need to update your memory_limit depending on the number of iterations
//Output the start memory
print 'Start: '.memory_get_usage()."B
Below test results are in MB
";
//Our 1MB string
global $OneMB, $NumIterations;
$OneMB=str_repeat('x', 1024*1024);
$NumIterations=500;
//Run the tests
$ConcatTest=RunTest('ConcatTest');
$ImplodeTest=RunTest('ImplodeTest');
$RecurseTest=RunTest('RecurseTest');
//Output the results in a table
OutputResults(
Array('ConcatTest', 'ImplodeTest', 'RecurseTest'),
Array($ConcatTest, $ImplodeTest, $RecurseTest)
);
//Start a test run by initializing the array that will hold the results and manipulating those results after the test is complete
function RunTest($TestName)
{
$CurrentTestNums=Array();
$TestStartMem=memory_get_usage();
$StartTime=microtime(true);
RunTestReal($TestName, $CurrentTestNums, $StrLen);
$CurrentTestNums[]=memory_get_usage();
//Subtract $TestStartMem from all other numbers
foreach($CurrentTestNums as &$Num)
$Num-=$TestStartMem;
unset($Num);
$CurrentTestNums[]=$StrLen;
$CurrentTestNums[]=microtime(true)-$StartTime;
return $CurrentTestNums;
}
//Initialize the test and store the memory allocated at the end of the test, with the result
function RunTestReal($TestName, &$CurrentTestNums, &$StrLen)
{
$R=$TestName($CurrentTestNums);
$CurrentTestNums[]=memory_get_usage();
$StrLen=strlen($R);
}
//Concatenate 1MB string over and over onto a single string
function ConcatTest(&$CurrentTestNums)
{
global $OneMB, $NumIterations;
$Result='';
for($i=0;$i<$NumIterations;$i++)
{
$Result.=$OneMB;
$CurrentTestNums[]=memory_get_usage();
}
return $Result;
}
//Create an array of 1MB strings and then join w/ an implode
function ImplodeTest(&$CurrentTestNums)
{
global $OneMB, $NumIterations;
$Result=Array();
for($i=0;$i<$NumIterations;$i++)
{
$Result[]=$OneMB;
$CurrentTestNums[]=memory_get_usage();
}
return implode('', $Result);
}
//Recursively add strings onto each other
function RecurseTest(&$CurrentTestNums, $TestNum=0)
{
Global $OneMB, $NumIterations;
if($TestNum==$NumIterations)
return '';
$NewStr=RecurseTest($CurrentTestNums, $TestNum+1).$OneMB;
$CurrentTestNums[]=memory_get_usage();
return $NewStr;
}
//Output the results in a table
function OutputResults($TestNames, $TestResults)
{
global $NumIterations;
print '<table border=1 cellspacing=0 cellpadding=2><tr><th>Test Name</th><th>'.implode('</th><th>', $TestNames).'</th></tr>';
$FinalNames=Array('Final Result', 'Clean');
for($i=0;$i<$NumIterations+2;$i++)
{
$TestName=($i<$NumIterations ? $i : $FinalNames[$i-$NumIterations]);
print "<tr><th>$TestName</th>";
foreach($TestResults as $TR)
printf('<td>%07.4f</td>', $TR[$i]/1024/1024);
print '</tr>';
}
//Other result numbers
print '<tr><th>Final String Size</th>';
foreach($TestResults as $TR)
printf('<td>%d</td>', $TR[$NumIterations+2]);
print '</tr><tr><th>Runtime</th>';
foreach($TestResults as $TR)
printf('<td>%s</td>', $TR[$NumIterations+3]);
print '</tr></table>';
}
?>
30
« on: September 24, 2016, 02:40:27 am »
function DeepObjectCompare(O1, O2)
{
try {
DOC_Val(O1, O2, ['O1->O2', O1, O2]);
return DOC_Val(O2, O1, ['O2->O1', O1, O2]);
} catch(e) {
console.log(e.Chain);
throw(e);
}
}
function DOC_Error(Reason, Chain, Val1, Val2)
{
this.Reason=Reason;
this.Chain=Chain;
this.Val1=Val1;
this.Val2=Val2;
}
function DOC_Val(Val1, Val2, Chain)
{
function DoThrow(Reason, NewChain) { throw(new DOC_Error(Reason, NewChain!==undefined ? NewChain : Chain, Val1, Val2)); }
if(typeof(Val1)!==typeof(Val2))
return DoThrow('Type Mismatch');
if(Val1===null || Val1===undefined)
return Val1!==Val2 ? DoThrow('Null/undefined mismatch') : true;
if(Val1.constructor!==Val2.constructor)
return DoThrow('Constructor mismatch');
switch(typeof(Val1))
{
case 'object':
for(var m in Val1)
{
if(!Val1.hasOwnProperty(m))
continue;
var CurChain=Chain.concat([m]);
if(!Val2.hasOwnProperty(m))
return DoThrow('Val2 missing property', CurChain);
DOC_Val(Val1[m], Val2[m], CurChain);
}
return true;
case 'number':
if(Number.isNaN(Val1))
return !Number.isNaN(Val2) ? DoThrow('NaN mismatch') : true;
case 'string':
case 'boolean':
return Val1!==Val2 ? DoThrow('Value mismatch') : true;
case 'function':
if(Val1.prototype!==Val2.prototype)
return DoThrow('Prototype mismatch');
if(Val1!==Val2)
return DoThrow('Function mismatch');
return true;
default:
return DoThrow('Val1 is unknown type');
}
}
|
|
