Additional string parameters that have %THEVAR% are replaced with the value being checked
If the first item is NULL, it will be removed, and the return will include full rows ("*") of the query result set
Updated functionality:
json_encode() In RetMsg() and CallByAction() now use JSON_UNESCAPED_UNICODE
GetVars.VarArray.IsOptional now only triggers if it is (boolean)true
Added additional description specification to GetVars.VarArray.SQLLookup which says it uses the additional parameters as values to fill in the SQL Query
Added static member $InitialPrintAndDieOnError which DSQL.PrintAndDieOnError inherits on creation
Bug Fixes:
mysqli_set_charset is set to utf-8
In FormatSQLError() the date() function used for “Start Time” now uses “24-hour format of an hour with leading zeros” [date(“H”)] instead of “12-hour format of an hour without leading zeros” [date(“g”)]
Following is some C++ source code for a Windows kernel-driver service loader. It could be used to load other service types too by changing the dwServiceType flag on the CreateService call. I threw this together for another project I am currently working on. It is also used in the following post (posting soon).
It works in the following way:
It is a command line utility which takes 3 arguments:
The service name. Hereby referred to as SERVICE_NAME
The service display name. Hereby referred to as DISPLAY_NAME
The driver path (to the .sys file). Hereby referred to as DRIVER_PATH
This program (most likely) requires administrative access. There are also some caveats regarding driver code signing requirements that are thoroughly explored elsewhere.
It first checks to see if a service already exists with the given SERVICE_NAME. If it does:
If the DISPLAY_NAME matches, the service is kept as is.
If the DISPLAY_NAME does not match, the user is prompted on if they want to delete the current service. If they do not, the program exits.
If the service needs to be created (it did not already exist or was deleted), it creates the service with the given SERVICE_NAME, DISPLAY_NAME, and DRIVER_PATH. If the service is not created during this run, the DRIVER_PATH is ignored. Note: The DRIVER_PATH must be to a direct local file system file. I have found that network links and symbolic links do not work.
The service is started up:
If it is already running, the user is prompted on if they want to stop the currently running service. If they say no, the program exits.
The program then waits for a final user input on if they want to close the service before exiting the program.
If there was an error, the program reports the error, otherwise, it reports “Success”.
The program pauses at the end until the user presses any key to exit.
The program returns 0 on success, and 1 if an error occurred.
//Compiler flags #define WIN32_LEAN_AND_MEAN //Include minimum amount of windows stuff #ifndef _UNICODE //Everything in this script is unicode #define _UNICODE #endif
//Function declarations WCHAR* InitDriver(int argc, WCHAR *argv[]); WCHAR* FormatError(WCHAR* Format, ...); SmartWinAlloc GetLastErrorStr(); BOOLEAN AskQuestion(WCHAR* Question); //Returns if user answered yes
int wmain(int argc, WCHAR *argv[]) { //Run the init routine WCHAR* Ret=InitDriver(argc, argv);
//If there is an error, report it, or otherwise, report success wprintf(L"%s\n", Ret ? Ret : L"Success"); wprintf(L"%s\n", L"Press any key to exit"); _getch();
//Open the service manager wprintf(L"%s\n", L"Opening the service manager"); SC_HANDLE HSCManager=OpenSCManager(nullptr, nullptr, SC_MANAGER_CREATE_SERVICE); if(!HSCManager) return FormatError(L"%s: %s", L"Error opening service manager", GetLastErrorStr()); SmartCloseService FreeHSCManager(&HSCManager, Delete_SmartCloseService);
//Check if the service already exists wprintf(L"%s\n", L"Checking previously existing service state"); BOOL ServiceExists=false; { //Get the service name const DWORD NameBufferSize=255; WCHAR NameBuffer[NameBufferSize]; WCHAR *NamePointer=NameBuffer; DWORD NamePointerSize=NameBufferSize; std::unique_ptr<WCHAR> Buf(nullptr); //May be swapped with a real pointer later for(INT_PTR i=0;i<2;i++) { //If we found the service, exit the lookup here if(GetServiceDisplayName(HSCManager, Param_ServiceName, NamePointer, &NamePointerSize)) { ServiceExists=true; break; }
//If the service does not exist, we can exit the lookup here if(GetLastError()==ERROR_SERVICE_DOES_NOT_EXIST) break;
//If error is not insufficient buffer size, return the error if(GetLastError()!=ERROR_INSUFFICIENT_BUFFER) return FormatError(L"%s: %s", L"Could not query service information", GetLastErrorStr());
//If second pass, error out if(i==1) return FormatError(L"%s: %s", L"Could not query service information", L"Second buffer pass failed");
//Create a buffer of appropriate size (and make sure it will later be released) NamePointer=new WCHAR[++NamePointerSize]; std::unique_ptr<WCHAR> Buf2(NamePointer); Buf.swap(Buf2); }
//If the service already exists, confirm the service name matches, and if not, ask if user wants to delete the current service if(ServiceExists) { wprintf(L"%s\n", L"The service already exists"); if(wcsncmp(NamePointer, Param_DisplayName, NamePointerSize+1)) { //If the server names do not match, ask the user what to do wprintf(L"%s:\nCurrent: %s\nRequested: %s\n", L"The service names do not match", NamePointer, Param_DisplayName);
//Make the request if(!AskQuestion(L"Would you like to replace the service? (y/n)")) //If user does not wish to replace the service return FormatError(L"%s", L"Cannot continue if service names do not match");
//Delete the service wprintf(L"%s\n", L"Deleting the old service"); ServiceExists=false; SC_HANDLE TheService=OpenService(HSCManager, Param_ServiceName, DELETE); if(!TheService) return FormatError(L"%s: %s", L"Could not open the service to delete it", GetLastErrorStr()); SmartCloseService CloseTheService(&TheService, Delete_SmartCloseService); //Close the service handle if(!DeleteService(TheService)) return FormatError(L"%s: %s", L"Could not delete the service", GetLastErrorStr()); wprintf(L"%s\n", L"The service has been deleted"); } } }
//Create the service SC_HANDLE TheService; if(!ServiceExists) { //Confirm the driver path exists wprintf(L"%s\n", L"Checking the driver file"); DWORD FileAttrs=GetFileAttributes(Param_DriverPath); if(FileAttrs==INVALID_FILE_ATTRIBUTES) return FormatError(L"%s: %s", L"Given path is invalid", GetLastErrorStr()); if(FileAttrs&FILE_ATTRIBUTE_DIRECTORY) return FormatError(L"%s: %s", L"Given path is invalid", L"Path is a folder");
//Create the service wprintf(L"%s\n", L"Creating the service"); TheService=CreateService( HSCManager, Param_ServiceName, Param_DisplayName, SERVICE_START|SERVICE_STOP, SERVICE_KERNEL_DRIVER, SERVICE_DEMAND_START, SERVICE_ERROR_IGNORE, Param_DriverPath, nullptr, nullptr, nullptr, nullptr, nullptr); if(!TheService) return FormatError(L"%s: %s", L"Could not create the service", GetLastErrorStr());
//Open the service if not creating } else { TheService=OpenService(HSCManager, Param_ServiceName, SERVICE_START|SERVICE_STOP); if(!TheService) return FormatError(L"%s: %s", L"Could not open the service", GetLastErrorStr()); } SmartCloseService CloseTheService(&TheService, Delete_SmartCloseService); //Close the service on exit
//Start the service wprintf(L"%s\n", L"Starting the service"); for(INT_PTR i=0;i<2;i++) { if(StartService(TheService, 0, nullptr)) break;
//If not "service already running" error, or user does not want to stop the current service if(i==1 || GetLastError()!=ERROR_SERVICE_ALREADY_RUNNING || !AskQuestion(L"The service is already running. Would you like to stop it? (y/n)")) return FormatError(L"%s: %s", L"Could not start the service", GetLastErrorStr());
//Stop the service SERVICE_STATUS ss; wprintf(L"%s\n", L"Stopping the current service"); if(!ControlService(TheService, SERVICE_CONTROL_STOP, &ss)) return FormatError(L"%s: %s", L"Could not stop the current service", GetLastErrorStr()); } wprintf(L"%s\n", L"Started the service");
//Ask if the user wants to close the service if(!AskQuestion(L"Would you like to stop the service before exit? (y/n)")) return nullptr;
//Stop the service SERVICE_STATUS ss; if(!ControlService(TheService, SERVICE_CONTROL_STOP, &ss)) return FormatError(L"%s: %s", L"Could not stop the service", GetLastErrorStr()); if(ss.dwCurrentState!=SERVICE_STOP_PENDING && ss.dwCurrentState!=SERVICE_STOPPED) return FormatError(L"%s", L"The service does not appear to be closing"); wprintf(L"%s\n", L"The service has been stopped");
//Return success return nullptr; }
WCHAR* FormatError(WCHAR* Format, ...) { static WCHAR Err[255]; va_list VAList; va_start(VAList, Format); vswprintf(Err, sizeof(Err)/sizeof(Err[0]), Format, VAList); return Err; }
BOOLEAN AskQuestion(WCHAR* Question) { //Make the request and wait for an input character while(1) { //Ask the question and get the answer wprintf(L"%s:", Question); fflush(stdout); char InputChar=_getch(); printf("\n");
//Check for a valid answer if(InputChar=='n' || InputChar=='N') return FALSE; if(InputChar=='y' || InputChar=='Y') return TRUE; } }
Fixed absolute path logic for Linux (partial credit to Matt Spitz)
Bypasses UTF8 BOM
Program now works off of argument flags.
Added parameters: Playlist encoding, override type, force list
Created special BulletHelpFormatter class for parameters
Playlist names can now conflict with other item/list names in Plex
Console column width passthrough in the .sh file
Updated READMEs regarding:
Unicode compliance
The “no such module : FTS4” error
Running the script from a computer external to the server running Plex
The “The program can’t start because MSVCR100.dll is missing” error
All updates
Added the ability to compile to a windows executable (via setup.py py2exe)
The Playlist Name is now an optional argument which can be entered after the program is ran. This allows directly dragging playlists onto the executable
Added shebang to main script
M3U files now ignore lines that are empty or have only whitespace
The PHP MySQL extension is being deprecated in favor of the MySQLi extension in PHP 5.5, and removed as of PHP 7.0. MySQLi was first referenced in PHP v5.0.0 beta 4 on 2004-02-12, with the first stable release in PHP 5.0.0 on 2004-07-13[1]. Before that, the PHP MySQL extension was by far the most popular way of interacting with MySQL on PHP, and still was for a very long time after. This website was opened only 2 years after the first stable release!
With the deprecation, problems from some websites I help host have popped up, many of these sites being very, very old. I needed a quick and dirty solution to monkey-patch these websites to use MySQLi without rewriting all their code. The obvious answer is to overwrite the functions with wrappers for MySQLi. The generally known way of doing this is with the Advanced PHP Debugger (APD). However, using this extension has a lot of requirements that are not appropriate for a production web server. Fortunately, another extension I recently learned of offers the renaming functionality; runkit. It was a super simple install for me.
From the command line, run “pecl install runkit”
Add “extension=runkit.so” and “runkit.internal_override=On” to the php.ini
Besides the ability to override these functions with wrappers, I also needed a way to make sure this file was always loaded before all other PHP files. The simple solution for that is adding “auto_prepend_file=/PATH/TO/FILE” to the “.user.ini” in the user’s root web directory.
The code for this script is as follows. It only contains a limited set of the MySQL functions, including some very esoteric ones that the web site used. This is not a foolproof script, but it gets the job done.
//If a connection is not explicitely passed to a mysql_ function, use the last created connection global$SQLLink; //The remembered SQL Link functionGetConn($PassedConn) { if(isset($PassedConn)) return$PassedConn; global$SQLLink; return$SQLLink; }
def_split_lines(self, text, width): #Split lines around line breaks and then modify each line Lines=[] for Line in text.splitlines(): #Get the number of spaces to put at subsequent lines #0 if not a list item, oherwise, 2+list item start ListEl=re.match(r'^*\*', Line) NumBeginningSpace=(0if ListEl==Noneelse ListEl.end()+1)
#Add extra spaces at the beginning of each line to match the start of the current line, and go to a maxium of $width IsFirstPass=True SpacesToAdd='' NumSpacesToAdd=0 while(True): #Get the word break points before and after where the line would end MaxLineLen=max(min(width-NumSpacesToAdd, len(Line)), 1) PrevWordBreak=CurWordBreak=0 for WordBreak in re.finditer(r'(?<=\W).|\W|$', Line): PrevWordBreak=CurWordBreak CurWordBreak=WordBreak.start() if CurWordBreak>=MaxLineLen: if CurWordBreak==MaxLineLen: PrevWordBreak=CurWordBreak break
#If previous wordbreak is more than MinCharsInSplitWord away from MaxLineLen, then split at the end of the line IsSplit=(PrevWordBreak<1or CurWordBreak-PrevWordBreak>self.MinCharsInSplitWord) SplitPos=(MaxLineLen if IsSplit else PrevWordBreak)
#Append the new line to the list of lines Lines.append(SpacesToAdd+Line[0:SplitPos]+('-'if IsSplit else'')) Line=Line[SplitPos:]
#If this is the end, nothing left to do iflen(Line)==0: break
#If this is the first pass, update line creation variables if IsFirstPass: IsFirstPass=False NumSpacesToAdd=NumBeginningSpace SpacesToAdd=(''* NumSpacesToAdd)
I had the need for a project recently to have a database list of countries and currencies. I ended up pulling the currency list from wikipedia and the country list from the stage.gov website. You can download the SQL here, or copy from below. Do note there are NULLs in some of the places where data was unavailable.
DROP TABLE IF EXISTS Countries; CREATE TABLE Countries ( ID tinyint(3) unsigned NOT NULL AUTO_INCREMENT, FIPS char(2) CHARACTER SET ascii COLLATE ascii_bin NOT NULL, Name varchar(35) NOT NULL, LongName varchar(55) NOT NULL, PRIMARY KEY (ID), UNIQUE KEY FIPS (FIPS), UNIQUE KEY Name (Name) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
INSERT INTO Countries (FIPS, Name, LongName) VALUES ('AF','Afghanistan','Islamic Republic of Afghanistan'), ('AL','Albania','Republic of Albania'), ('AG','Algeria','People’s Democratic Republic of Algeria'), ('AN','Andorra','Principality of Andorra'), ('AO','Angola','Republic of Angola'), ('AC','Antigua and Barbuda','Antiqua and Barbuda'), ('AR','Argentina','Argentine Republic'), ('AM','Armenia','Republic of Armenia'), ('AS','Australia','Commonwealth of Australia'), ('AU','Austria','Republic of Austria'), ('AJ','Azerbaijan','Republic of Azerbaijan'), ('BF','Bahamas, The','Commonwealth of The Bahamas'), ('BA','Bahrain','Kingdom of Bahrain'), ('BG','Bangladesh','People’s Republic of Bangladesh'), ('BB','Barbados','Barbados'), ('BO','Belarus','Republic of Belarus'), ('BE','Belgium','Kingdom of Belgium'), ('BH','Belize','Belize'), ('BN','Benin','Republic of Benin'), ('BT','Bhutan','Kingdom of Bhutan'), ('BL','Bolivia','Plurinational State of Bolivia'), ('BK','Bosnia and Herzegovina','Bosnia and Herzegovina'), ('BC','Botswana','Republic of Botswana'), ('BR','Brazil','Federative Republic of Brazil'), ('BX','Brunei','Brunei Darussalam'), ('BU','Bulgaria','Republic of Bulgaria'), ('UV','Burkina Faso','Burkina Faso'), ('BM','Burma','Union of Burma'), ('BY','Burundi','Republic of Burundi'), ('CV','Cabo Verde','Republic of Cabo Verde'), ('CB','Cambodia','Kingdom of Cambodia'), ('CM','Cameroon','Republic of Cameroon'), ('CA','Canada','Canada'), ('CT','Central African Republic','Central African Republic'), ('CD','Chad','Republic of Chad'), ('CI','Chile','Republic of Chile'), ('CH','China','People’s Republic of China'), ('CO','Colombia','Republic of Colombia'), ('CN','Comoros','Union of the Comoros'), ('CF','Congo (Brazzaville)','Republic of the Congo'), ('CG','Congo (Kinshasa)','Democratic Republic of the Congo'), ('CS','Costa Rica','Republic of Costa Rica'), ('IV','Côte d’Ivoire','Republic of Côte d’Ivoire'), ('HR','Croatia','Republic of Croatia'), ('CU','Cuba','Republic of Cuba'), ('CY','Cyprus','Republic of Cyprus'), ('EZ','Czech Republic','Czech Republic'), ('DA','Denmark','Kingdom of Denmark'), ('DJ','Djibouti','Republic of Djibouti'), ('DO','Dominica','Commonwealth of Dominica'), ('DR','Dominican Republic','Dominican Republic'), ('EC','Ecuador','Republic of Ecuador'), ('EG','Egypt','Arab Republic of Egypt'), ('ES','El Salvador','Republic of El Salvador'), ('EK','Equatorial Guinea','Republic of Equatorial Guinea'), ('ER','Eritrea','State of Eritrea'), ('EN','Estonia','Republic of Estonia'), ('ET','Ethiopia','Federal Democratic Republic of Ethiopia'), ('FJ','Fiji','Republic of Fiji'), ('FI','Finland','Republic of Finland'), ('FR','France','French Republic'), ('GB','Gabon','Gabonese Republic'), ('GA','Gambia, The','Republic of The Gambia'), ('GG','Georgia','Georgia'), ('GM','Germany','Federal Republic of Germany'), ('GH','Ghana','Republic of Ghana'), ('GR','Greece','Hellenic Republic'), ('GJ','Grenada','Grenada'), ('GT','Guatemala','Republic of Guatemala'), ('GV','Guinea','Republic of Guinea'), ('PU','Guinea-Bissau','Republic of Guinea-Bissau'), ('GY','Guyana','Co-operative Republic of Guyana'), ('HA','Haiti','Republic of Haiti'), ('VT','Holy See','Holy See'), ('HO','Honduras','Republic of Honduras'), ('HU','Hungary','Hungary'), ('IC','Iceland','Republic of Iceland'), ('IN','India','Republic of India'), ('ID','Indonesia','Republic of Indonesia'), ('IR','Iran','Islamic Republic of Iran'), ('IZ','Iraq','Republic of Iraq'), ('EI','Ireland','Ireland'), ('IS','Israel','State of Israel'), ('IT','Italy','Italian Republic'), ('JM','Jamaica','Jamaica'), ('JA','Japan','Japan'), ('JO','Jordan','Hashemite Kingdom of Jordan'), ('KZ','Kazakhstan','Republic of Kazakhstan'), ('KE','Kenya','Republic of Kenya'), ('KR','Kiribati','Republic of Kiribati'), ('KN','Korea, North','Democratic People’s Republic of Korea'), ('KS','Korea, South','Republic of Korea'), ('KV','Kosovo','Republic of Kosovo'), ('KU','Kuwait','State of Kuwait'), ('KG','Kyrgyzstan','Kyrgyz Republic'), ('LA','Laos','Lao People’s Democratic Republic'), ('LG','Latvia','Republic of Latvia'), ('LE','Lebanon','Lebanese Republic'), ('LT','Lesotho','Kingdom of Lesotho'), ('LI','Liberia','Republic of Liberia'), ('LY','Libya','Libya'), ('LS','Liechtenstein','Principality of Liechtenstein'), ('LH','Lithuania','Republic of Lithuania'), ('LU','Luxembourg','Grand Duchy of Luxembourg'), ('MK','Macedonia','Republic of Macedonia'), ('MA','Madagascar','Republic of Madagascar'), ('MI','Malawi','Republic of Malawi'), ('MY','Malaysia','Malaysia'), ('MV','Maldives','Republic of Maldives'), ('ML','Mali','Republic of Mali'), ('MT','Malta','Republic of Malta'), ('RM','Marshall Islands','Republic of the Marshall Islands'), ('MR','Mauritania','Islamic Republic of Mauritania'), ('MP','Mauritius','Republic of Mauritius'), ('MX','Mexico','United Mexican States'), ('FM','Micronesia, Federated States of','Federated States of Micronesia'), ('MD','Moldova','Republic of Moldova'), ('MN','Monaco','Principality of Monaco'), ('MG','Mongolia','Mongolia'), ('MJ','Montenegro','Montenegro'), ('MO','Morocco','Kingdom of Morocco'), ('MZ','Mozambique','Republic of Mozambique'), ('WA','Namibia','Republic of Namibia'), ('NR','Nauru','Republic of Nauru'), ('NP','Nepal','Federal Democratic Republic of Nepal'), ('NL','Netherlands','Kingdom of the Netherlands'), ('NZ','New Zealand','New Zealand'), ('NU','Nicaragua','Republic of Nicaragua'), ('NG','Niger','Republic of Niger'), ('NI','Nigeria','Federal Republic of Nigeria'), ('NO','Norway','Kingdom of Norway'), ('MU','Oman','Sultanate of Oman'), ('PK','Pakistan','Islamic Republic of Pakistan'), ('PS','Palau','Republic of Palau'), ('PM','Panama','Republic of Panama'), ('PP','Papua New Guinea','Independent State of Papua New Guinea'), ('PA','Paraguay','Republic of Paraguay'), ('PE','Peru','Republic of Peru'), ('RP','Philippines','Republic of the Philippines'), ('PL','Poland','Republic of Poland'), ('PO','Portugal','Portuguese Republic'), ('QA','Qatar','State of Qatar'), ('RO','Romania','Romania'), ('RS','Russia','Russian Federation'), ('RW','Rwanda','Republic of Rwanda'), ('SC','Saint Kitts and Nevis','Federation of Saint Kitts and Nevis'), ('ST','Saint Lucia','Saint Lucia'), ('VC','Saint Vincent and the Grenadines','Saint Vincent and the Grenadines'), ('WS','Samoa','Independent State of Samoa'), ('SM','San Marino','Republic of San Marino'), ('TP','Sao Tome and Principe','Democratic Republic of Sao Tome and Principe'), ('SA','Saudi Arabia','Kingdom of Saudi Arabia'), ('SG','Senegal','Republic of Senegal'), ('RI','Serbia','Republic of Serbia'), ('SE','Seychelles','Republic of Seychelles'), ('SL','Sierra Leone','Republic of Sierra Leone'), ('SN','Singapore','Republic of Singapore'), ('LO','Slovakia','Slovak Republic'), ('SI','Slovenia','Republic of Slovenia'), ('BP','Solomon Islands','Solomon Islands'), ('SO','Somalia','Federal Republic of Somalia'), ('SF','South Africa','Republic of South Africa'), ('OD','South Sudan','Republic of South Sudan'), ('SP','Spain','Kingdom of Spain'), ('CE','Sri Lanka','Democratic Socialist Republic of Sri Lanka'), ('SU','Sudan','Republic of the Sudan'), ('NS','Suriname','Republic of Suriname'), ('WZ','Swaziland','Kingdom of Swaziland'), ('SW','Sweden','Kingdom of Sweden'), ('SZ','Switzerland','Swiss Confederation'), ('SY','Syria','Syrian Arab Republic'), ('TI','Tajikistan','Republic of Tajikistan'), ('TZ','Tanzania','United Republic of Tanzania'), ('TH','Thailand','Kingdom of Thailand'), ('TT','Timor-Leste','Democratic Republic of Timor-Leste'), ('TO','Togo','Togolese Republic'), ('TN','Tonga','Kingdom of Tonga'), ('TD','Trinidad and Tobago','Republic of Trinidad and Tobago'), ('TS','Tunisia','Republic of Tunisia'), ('TU','Turkey','Republic of Turkey'), ('TX','Turkmenistan','Turkmenistan'), ('TV','Tuvalu','Tuvalu'), ('UG','Uganda','Republic of Uganda'), ('UP','Ukraine','Ukraine'), ('AE','United Arab Emirates','United Arab Emirates'), ('UK','United Kingdom','United Kingdom of Great Britain and Northern Ireland'), ('US','United States','United States of America'), ('UY','Uruguay','Oriental Republic of Uruguay'), ('UZ','Uzbekistan','Republic of Uzbekistan'), ('NH','Vanuatu','Republic of Vanuatu'), ('VE','Venezuela','Bolivarian Republic of Venezuela'), ('VM','Vietnam','Socialist Republic of Vietnam'), ('YM','Yemen','Republic of Yemen'), ('ZA','Zambia','Republic of Zambia'), ('ZI','Zimbabwe','Republic of Zimbabwe');
DROP TABLE IF EXISTS Currencies; CREATE TABLE Currencies ( ID tinyint(3) unsigned NOT NULL AUTO_INCREMENT, ISOCode char(3) CHARACTER SET ascii COLLATE ascii_bin DEFAULT NULL, NumberToBasic smallint(5) unsigned DEFAULT NULL, Symbol varchar(6) DEFAULT NULL, Name varchar(55) DEFAULT NULL, PRIMARY KEY (ID), UNIQUE KEY ISOCode (ISOCode), UNIQUE KEY Name (Name) ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
So somehow all of the file names in my Rammstein music directory, and some in my Daft Punk, had characters with diacritics replaced with an invalid character. I pasted one of such filenames into a hex editor to evaluate what the problem was. First, I should note that Windows encodes its filenames (and pretty much everything) in UTF16. Everything else in the world (mostly) has settled on UTF8, which is a much better encoding for many reasons. So during some file copy/conversion at some point in the directories’ lifetime, the file names had done a freakish (utf16*)(utf16->utf8) rename, or something to that extent. I had noticed that all I needed to do was to replace the first 2 bytes of the diacritic character with a different byte. Namely “EF 8x” to “Cx”, and the rest of the bytes for the character were fine. So if anyone ever needs it, here is the bash script.
LANG=; IFS=$'\n' for i in `find -type f | grep -P '\xEF[\x80-\x8F]'`; do FROM="$i"; TO=$(echo "$i" | perl -pi -e 's/\xEF([\x80-\x8F])/pack("C", ord($1)+(0xC0-0x80))/e'); echo Renaming "'$FROM'" to "'$TO'" mv "$FROM" "$TO" done
I may need to expand the range beyond the x80-x8F range, but am unsure at this point. I only confirmed the range x82-x83.
While I’ve been encountering more bugs than I can count on both hands while working with XCode, this one takes the cake. Clang (the compiler) was throwing the following errors while it was trying to compile one of its objective C source files (.m extension).
clang: error: unable to execute command: Segmentation fault: 11 clang: error: clang frontend command failed due to signal (use -v to see invocation) Apple LLVM version 7.0.2 (clang-700.1.81) Target: arm-apple-darwin14.5.0 Thread model: posix clang: note: diagnostic msg: PLEASE submit a bug report to http://developer.apple.com/bugreporter/ and include the crash backtrace, preprocessed source, and associated run script. clang: error: unable to execute command: Segmentation fault: 11 clang: note: diagnostic msg: Error generating preprocessed source(s).
The fix... was to keep the specific source file open in an XCode window ~.~ . How the heck do you integrate the [CLI] compiler so much into the IDE that this could happen? Or is this simply a weird file system thing? I should note that my XCode project directory, with all files, is located on a VMware volume share.
One of the companies I work for recently moved one of our cPanel servers to a new collocation, still running cPanel. We decided to use a new backup solution called r1soft, which so far has been working spectacularly. I’d love to use it for my personal computers, except the licenses, which are geared towards enterprise business, are way too costly.
However, since r1soft only backs up files (on the incrementally block level, yay) you can’t use it to restore a cPanel account. It can only restore things like the user’s home directory and SQL databases. Because of this, when we had need to restore an entire account today, and found out there is no easy/quick way to do it, we were up a creek. The obvious future solution for this would be to use cPanel’s backup (or legacy backup) systems, but unfortunately, you can’t easily set them to not backup the user’s databases and home directory, which can be very large, and are already taken care of by r1soft. I ended up adding the following script, ran nightly via cron, to back up user account settings.
It saves all the user settings under the backup path in their own directory, uncompressed, and named cpmove-USERNAME. It is best to do it this way so r1soft’s incremental backups don’t have much extra work if anything changes. Make sure to change line 3 in the following script to the path where you want backups to occur.
#!/bin/bash #Create and move to backup directory BACKUPDIR=/backup/userbackup mkdir -p $BACKUPDIR#Make sure the directory exists cd$BACKUPDIR
#Extract from and remove the tar container file tar -xvf cpmove-$USER.tar rm -f cpmove-$USER.tar
#Save MySQL user settings mysqldump --compact -fnt -w "User LIKE '$USER""_%'" mysql user db tables_priv columns_priv procs_priv proxies_priv \ | perl -pe "s~('|NULL)\),\('~\1),\n('~ig" \ > cpmove-$USER/mysql-users.sql done;
This script skips a few backup items that need to be noted. Mailman, logs, homedir, and bandwidth data should all be easy 1:1 copy over restores from r1soft. I excluded them because those can take up a lot of room, which we want r1soft to handle. The same goes for MySQL, except that your MySQL users are not backed up to your account, which is why I added the final section.
Do note, for the final section, the line starting with “| perl” is optional. It is there to separate the insert rows into their own lines. A very minor warning though; it would also pick up cases where the last field in MySQL’s user table ends in “NULL),(”. This would only happen if someone is trying to be malicious and knew about this script, and even then, it couldn’t harm anything.
Bonus note: To restore a MySQL database which does not use a shared-file (like InnoDB does by default), you could actually stop the MySQL server, copy over the binary database files, and start the server back up.
When I first created my website 10 years ago, from scratch, I did not want to deal with writing a comment system with HTML markups. And in those days, there weren’t plugins for everything like there is today. My solution was setting up a forum which would contain a topic for every Project, Update, and Post, and have my pages mirror the linked topic’s posts.
I had just put in a quick hack at the time in which the pulled SMF message’s body had links converted from bbcode (there might have been 1 other bbcode I also hooked). I had done this with regular expressions, which was a nasty hack.
So anywho, I finally got around to writing a script that converts SMF messages’ bbcode to HTML and caches it. You can download it here, or see the code below. The script is optimized so that it only ever needs to load SMF code when a post has not yet been cached. Caching happens during the initial loading of an SMF post within the script’s main function, and is discarded if the post is changed.
The script requires that you run the query on line #3 of itself in your SMF database. Directly after that are 3 variables you need to set. The script assumes you are already logged in to the appropriate user. To use it, call “GFTP\GetForumTopicPosts($ForumTopicID)”. I have the functions split up so you can do individual posts too if needed (requires a little extra code).
<? //This SQL command must be ran before using the script //ALTER TABLE smf_messages ADD body_html text, ADD body_md5 char(32) DEFAULT NULL;
functionGetForumTopicPosts($ForumTopicID) { //Change to the forum database global$ForumInfo; $CurDB=mysql_fetch_row(mysql_query('SELECT database()'))[0]; if($CurDB!=$ForumInfo['DBName']) mysql_select_db($ForumInfo['DBName']); $OldEncoding=SetEncoding(true);
//Get the posts $PostsInfos=Array(); $PostsQuery=mysql_query('SELECT'.implode(', ', PostFields())." FROM $ForumInfo[MessageTableName] WHERE id_topic='".intval($ForumTopicID). "' AND approved=1 ORDER BY id_msg ASC LIMIT 1, 9999999"); if($PostsQuery) //If query failed, do not process while(($PostInfo=mysql_fetch_assoc($PostsQuery)) && ($PostsInfos[]=$PostInfo)) if(md5($PostInfo['body'])!=$PostInfo['body_md5']) //If the body md5s do not match, get new value, otherwise, use cached value ProcessPost($PostsInfos[count($PostsInfos)-1]); //Process the lastest post as a reference
//Restore from the forum database if($CurDB!=$ForumInfo['DBName']) mysql_select_db($CurDB); SetEncoding(false, $OldEncoding);
//Return the posts return$PostsInfos; }
functionProcessPost(&$PostInfo) //PostInfo must have fields id_msg, body, body_md5, and body_html { //Load SMF global$ForumInfo; if(!defined('SMF')) { global$context; require_once(rtrim($ForumInfo['Location'], DIRECTORY_SEPARATOR).DIRECTORY_SEPARATOR.'SSI.php'); mysql_select_db($ForumInfo['DBName']); SetEncoding(); }
//Update the cached body_html field $ParsedCode=$PostInfo['body_html']=parse_bbc($PostInfo['body']); $EscapedHTMLBody=mysql_escape_string($ParsedCode); $BodyMD5=md5($PostInfo['body']); mysql_query("UPDATE$ForumInfo[MessageTableName] SET body_html='$EscapedHTMLBody', body_md5='$BodyMD5'WHERE id_msg=$PostInfo[id_msg]"); }
//The fields to select in the Post query functionPostFields() { returnArray('id_msg', 'poster_time', 'id_member', 'subject', 'poster_name', 'body', 'body_md5', 'body_html'); }
//Swap character encodings. Needs to be set to utf8 functionSetEncoding($GetOld=false, $NewSet=Array('utf8', 'utf8', 'utf8')) { //Get the old charset if required $CharacterVariables=Array('character_set_client', 'character_set_results', 'character_set_connection'); $OldSet=Array(); if($GetOld) { //Fill in variables with default in case they are not found foreach($CharacterVariablesas$Index=>$Variable) $OldSet[$Variable]='utf8';
//Query for the character sets and update the OldSet array $Query=mysql_query('SHOW VARIABLES LIKE "character_%"'); while($VariableInfo=mysql_fetch_assoc($Query)) if(isset($OldSet[$VariableInfo['Variable_name']])) $OldSet[$VariableInfo['Variable_name']]=$VariableInfo['Value'];
$OldSet=array_values($OldSet); //Turn back into numerical array }
//Change to the new database encoding $CompiledSets=Array(); foreach($CharacterVariablesas$Index=>$Variable) $CompiledSets[$Index]=$CharacterVariables[$Index].'="'.mysql_escape_string($NewSet[$Index]).'"'; mysql_query('SET '.implode(', ', $CompiledSets));
//If requested, return the previous values return$OldSet; } ?>
Amazon EC2 is a great resource for cheap virtual servers to do simple things, like DNS or (low bandwidth) VPNs. I had the need this morning to set up a DNS server for a company which needed to blacklist a list of domains. The simplest way to do this is by editing all the computers’ hostfiles, but that method leaves a lot to be desired. Namely, blocking entire domains (as opposed to single subdomains), and deploying changes. Centralizing in a single place makes the job instant, immediate, and in the end, faster.
The following are the steps I used to set this up on an EC2 server. All command line instructions are followed by a single command you can run to execute the step. There is a full script below, at the end of the post, containing all steps from when you first login to SSH ("Login to root") to the end.
I am not going to go into the details of setting up an EC2 instance, as that information can be found elsewhere. I will also be skipping over some of the more obvious steps. Just create a default EC2 instance with the “Amazon Linux AMI”, and I will list all the changes that need to be made beyond that.
Creating the instance
For the first year, for the instance type, you might as well use a t2.micro, as it is free. After that, a t2.nano (which is a new lower level) currently at $56.94/year ($0.0065/Hour), should be fine.
After you select your instance type, click “Review and Launch” to launch the instance with all of the defaults.
After the confirmation screen, it will ask you to create a key pair. You can see other tutorials about this and how it enables you to log into your instance.
Edit the security group
Next, you need to edit the security group for your instance to allow incoming connections.
Go to “Instances” under the “Instances” group on the left menu, and click your instance.
In the bottom of the window, in the “Descriptions” tab, click the link next to “Security Groups”, which will bring you to the proper group in the security groups tab.
Right click it and “Edit inbound Rules”.
Make sure it has the following rules with Source=Anywhere: ALL ICMP [For pinging], SSH, HTTP, DNS (UDP), DNS (TCP)
Assign a permanent IP to your instance
To do this, click the “Elastic IPs” under “Network & Security” in the left menu.
Click “Allocate New Address”.
After creating it, right click the new address, then “Associate Address”, and assign it to your new instance.
You should probably set this IP up as an A record somewhere. I will refer to this IP as dns.yourdomain.com from now on.
Login to root
SSH into your instance as the ec2-user via “ssh ec2-user@dns.yourdomain.com”. If in windows, you could also use putty.
Sudo into root via “sudo su”.
Allow root login
At this point, I recommend setting it up so you can directly root into the server. Warning: some people consider this a security risk.
Copy your key pair(s) to the root user via “cat /home/ec2-user/.ssh/authorized_keys > /root/.ssh/authorized_keys”
Set SSHD to permit root logins by changing the PermitRootLogin variable to “yes” in /etc/ssh/sshd_config. A quick command to do this is “perl -pi -e 's/^\s*#?\s*PermitRootLogin.*$/PermitRootLogin yes/igm' /etc/ssh/sshd_config”, and then reload the SSHD config with “service sshd reload”. Make sure to attempt to directly log into SSH as root before exiting your current session to make sure you haven’t locked yourself out.
Install apache (the web server), bind/named (the DNS server), and PHP (a scripting language)
yum -y install bind httpd php
Start and set services to run at boot
service httpd start; service named start; chkconfig httpd on; chkconfig named on;
Set the DNS server to be usable by other computers
Edit /etc/named.conf and change the 2 following lines to have the value “any”: “listen-on port 53” and “allow-query”
perl -pi -e 's/^(\s*(?:listen-on port 53|allow-query)\s*{).*$/$1 any; };/igm' /etc/named.conf; service named reload;
Point the DNS server to the blacklist files
This is done by adding “include "/var/named/blacklisted.conf";” to /etc/named.conf
Put the following into /var/named/blacklisted.db . Make sure to change dns.yourdomain.com to your domain (or otherwise, “localhost”), and 1.1.1.1 to dns.yourdomain.com’s (your server’s) IP address. Make sure to keep all periods intact.
$TTL 14400 @ IN SOA dns.yourdomain.com. dns.yourdomain.com ( 2003052800 86400 300 604800 3600 ) @ IN NS dns.yourdomain.com. @ IN A 1.1.1.1 * IN A 1.1.1.1
The first 2 lines tell the server the domains belong to it. The 3rd line sets the base blacklisted domain to your server’s IP. The 4th line sets all subdomains of the blacklisted domain to your server’s IP.
This can be done via (Update the first line with your values)
YOURDOMAIN="dns.yourdomain.com"; YOURIP="1.1.1.1"; echo -ne "\$TTL 14400\n@ IN SOA $YOURDOMAIN. $YOURDOMAIN ( 2003052800 86400 300 604800 3600 )\n@ IN NS $YOURDOMAIN.\n@ IN A $YOURIP\n* IN A $YOURIP" > /var/named/blacklisted.db;
Fix the permissions on the blacklist files
chgrp named /var/named/blacklisted.*; chmod 660 /var/named/blacklisted.*;
Set the server’s domain resolution name servers
The server always needs to look at itself before other DNS servers. To do this, comment out everything in /etc/resolv.conf and add to it “nameserver localhost”. This is not the best solution. I’ll find something better later.
At this point, it’s a good idea to make sure the DNS server is working as intended. So first, we’ll add an example domain to the DNS server.
Add the following to /var/named/blacklisted.conf and restart named to get the server going with example.com: “zone "example.com" { type master; file "blacklisted.db"; };”
echo 'zone "example.com" { type master; file "blacklisted.db"; };' >> /var/named/blacklisted.conf; service named reload;
Ping “test.example.com” and make sure it’s IP is your server’s IP
Set your computer’s DNS to your server’s IP in your computer’s network settings, ping “test.example.com” from your computer, and make sure the returned IP is your server’s IP. If it works, you can restore your computer’s DNS settings.
Have the server return a message when a blacklisted domain is accessed
Add your message to /var/www/html
echo 'Domain is blocked' > /var/www/html/index.html
Set all URL paths to show the message by adding the following to the /var/www/html/.htaccess file
Turn on AllowOverride in the /etc/httpd/conf/httpd.conf for the document directory (/var/www/html/) via “ perl -0777 -pi -e 's~(<Directory "/var/www/html">.*?\n\s*AllowOverride).*?\n~$1 All~s' /etc/httpd/conf/httpd.conf”
Start the server via “service httpd graceful”
Create a script that allows apache to refresh the name server’s settings
Create a script at /var/www/html/AddRules/restart_named with “/sbin/service named reload” and set it to executable
Allow the user to run the script as root by adding to /etc/sudoers “apache ALL=(root) NOPASSWD: /var/www/html/AddRules/restart_named” and “Defaults!/var/www/html/AddRules/restart_named !requiretty”
//Output to the file $FinalDomainData=Array(); foreach($FinalDomainListas$Domain) $FinalDomainData[]= "zone \"$Domain\" { type master; file \"blacklisted.db\"; };"; file_put_contents($BlockedFile, implode("\n", $FinalDomainData));
//Reload named print`sudo /var/www/html/AddRules/restart_named`; ?>
Add the “apache” user to the “named” group so the script can update the list of domains in /var/named/blacklisted.conf via “usermod -a -G named apache; service httpd graceful;”
Run the domain update script
To add a domain (separate by commas): http://dns.yourdomain.com/AddRules/?Domains=domain1.com,domain2.com
To remove a domain (add “Remove&” after the “?”): http://dns.yourdomain.com/AddRules/?Remove&Domains=domain1.com,domain2.com
To list the domains: http://dns.yourdomain.com/AddRules/?List
Warning: Putting the password file in an http accessible directory is a security risk. I just did this for sake of organization.
Create the user+password via “htpasswd -bc /var/www/html/AddRules/.htpasswd USERNAME” and then entering the password
[Edit on 2016-01-30 @ noon]
To permanently set “localhost” as the resolver DNS, add “DNS1=localhost” to “/etc/sysconfig/network-scripts/ifcfg-eth0”. I have not yet confirmed this edit.
Security Issue
Soon after setting up this DNS server, it started getting hit by a DNS amplification attack. As the server is being used as a client’s DNS server, turning off recursion is not available. The best solution is to limit the people who can query the name server via an access list (usually a specific subnet), but that would very often not be an option either. The solution I currently have in place, which I have not actually verified if it works, is to add a forced-forward rule which only makes external requests to the name server provided by Amazon. To do this, get the name server’s IP from /etc/resolv.conf (it should be commented from an earlier step). Then add the following to your named.conf in the “options” section.
forwarders { DNS_SERVER_IP; }; forward only;
After I added this rule, external DNS requests stopped going through completely. To fix this, I turned “dnssec-validation” to “no” in the named.conf. Don’t forget to restart the service once you have made your changes.
Make sure to run this as root (login as root or sudo it)
Download the script here. Make sure to chmod and sudo it when running. “chmod +x dnsblacklist_install.sh; sudo ./dnsblacklist_install.sh;”
#User defined variables VARIABLES_SET=0; #Set this to 1 to allow the script to run YOUR_DOMAIN="localhost"; YOUR_IP="1.1.1.1"; BLOCKED_ERROR_MESSAGE="Domain is blocked"; ADDRULES_USERNAME="YourUserName"; ADDRULES_PASSWORD="YourPassword";
#Confirm script is ready to run if [ $VARIABLES_SET != 1 ];then echo'Variables need to be set in the script'; exit 1; fi if [ `whoami`!='root' ];then echo'Must be root to run script. When running the script, add "sudo" before it to' \ 'run as root'; exit 1; fi
#Install services yum -y install bind httpd php; chkconfig httpd on; chkconfig named on; service httpd start; service named start;
#Set the DNS server to be usable by other computers perl -pi -e 's/^(\s*(?:listen-on port 53|allow-query)\s*{).*$/$1 any; };/igm' \ /etc/named.conf; service named reload;
#Create the blacklist zone file echo -ne "\$TTL 14400 @ IN SOA $YOUR_DOMAIN. $YOUR_DOMAIN ( 2003052800 86400 300 604800 3600 ) @ IN NS $YOUR_DOMAIN. @ IN A $YOUR_IP * IN A $YOUR_IP"> /var/named/blacklisted.db;
#Fix the permissions on the blacklist files chgrp named /var/named/blacklisted.*; chmod 660 /var/named/blacklisted.*;
#Set the server’s domain resolution name servers perl -pi -e 's/^(?!;)/;/gm' /etc/resolv.conf; echo -ne '\nnameserver localhost'>> /etc/resolv.conf;
#Run a test echo'zone "example.com" { type master; file "blacklisted.db"; };'>> \ /var/named/blacklisted.conf; service named reload; FOUND_IP=`dig -t A example.com | grep -ioP "^example\.com\..*?"'in\s+a\s+[\d\.:]+'| \ grep -oP '[\d\.:]+$'`; if [ "$YOUR_IP"=="$FOUND_IP" ] then echo'Success: Example domain matches your given IP'> /dev/stderr; else echo'Warning: Example domain does not match your given IP'> /dev/stderr; fi
#Have the server return a message when a blacklisted domain is accessed echo"$BLOCKED_ERROR_MESSAGE"> /var/www/html/index.html; perl -0777 -pi -e 's~(<Directory "/var/www/html">.*?\n\s*AllowOverride).*?\n~$1 All~s' \ /etc/httpd/conf/httpd.conf; echo -n 'RewriteEngine on RewriteCond %{REQUEST_URI} !index.html RewriteCond %{REQUEST_URI} !AddRules/ RewriteRule ^(.*)$ /index.html [L]'> /var/www/html/.htaccess; service httpd graceful;
#Create a script that allows apache to refresh the name server’s settings mkdir /var/www/html/AddRules; echo'/sbin/service named reload'> /var/www/html/AddRules/restart_named; chmod 755 /var/www/html/AddRules/restart_named;
#Create a script that allows the user to add, remove, and list the blacklisted domains echo -n $'<?php //Get old domains $BlockedFile=\'/var/named/blacklisted.conf\'; $CurrentZones=Array(); foreach(explode("\\n", file_get_contents($BlockedFile)) as $Line) if(preg_match(\'/^zone "([\\w\\._-]+)"/\', $Line, $Results)) $CurrentZones[]=$Results[1];
//Output to the file $FinalDomainData=Array(); foreach($FinalDomainList as $Domain) $FinalDomainData[]="zone \\"$Domain\\" { type master; file \\"blacklisted.db\\"; };"; file_put_contents($BlockedFile, implode("\\n", $FinalDomainData));
//Reload named print `sudo /var/www/html/AddRules/restart_named`; ?>'> /var/www/html/AddRules/index.php;
usermod -a -G named apache; service httpd graceful;
