Import data with mysql using Load DATA INFILE tips

  • When importing hex string data, mysql load data will import them successfully but treat them as string without 0x indicating it as a hex string. For example, the imported result seen below: (description column, type longblob)
MyTable
pid  count  description(longblob datatype)
1,   1,     123456FF
2,   1,     aabbcc
3,   1,     ddeeff
instead of 

1,   1,     0x123456FF
2,   1,     0xaabbcc
3,   1,     0xddeeff

And even we manually add 0x to these hex string, when viewing it in mysql's Open Value in Editor, it shows them as literal string and if we save it to file, it is a literal string instead of the actual value the hex string represent so how do we resolve this issue.

1- we can use SET within Load Data Infile:
https://dev.mysql.com/doc/refman/8.0/en/load-data.html 
And We can fix this issue by assigning the column 'description' to a variable, and then set it to the UNHEXed version of that variable:
LOAD DATA INFILE 'MyFile.csv' INTO TABLE `MyTable` Fields Terminated by ',' Lines Terminated by '\n' (pid, count, @var1) SET description = UNHEX(@var1);

2- Or another easier method is once data imported, we can execute update statement on that column:

update MyTable set descritpion = UNHEX(description);
  • DateTime data either emptry string or null will be imported as ‘0000-00-00 00:00:00.000’ instead of default value null set in the table schema

The issue with this is because mysql set sql_mode to NO_ZERO_DATE.

To fix this issue, we can set global sql_mode to ALLOW_INVALID_DATES by executing a query below:

SET GLOBAL SQL_MODE='ALLOW_INVALID_DATES';
update MyTable set MyColumnDateTime = null where MyColumnDateTime  = '0000-00-00 00:00:00.000';
Be aware:
Sometimes with Load Data Infile, Uniqueidentify string aka guid e.g '2D3299C9-2CEA-48BE-BF4F-1115DA0CDCC5' will be imported in mysql varchar(64) column as small letter case '2d3299c9-2cea-48be-bf4f-1115da0cdcc5'
Be aware:                
Load Data Infile vs Load Data Local Infile

LOAD DATA INFILE gets the file from the database server's local filesystem. The file has to be located in the database directory or have world read permissions, and the client username must have the FILE privilege.

LOAD DATA LOCAL INFILE reads the file on the client, and sends the contents to the server.
When using Load Data Local Infile you will have to specify option '--local-infile=1' in the mysql commandline:
e.g: 
mysql --local-infile=1 -u MyUser -PMyPwd MyDB -e " Load Data Local Infile 'MyFile.csv' Into Table   'MyTable' Fields Terminated by ',' Lines Terminated by '\n'; "
Be aware: 
Bit column field
Load Data  infile from text into Bit type column: 
non empty string will be consider as bit 1 so even '0' from witin text file will be inserted into mysql by Load Data Infile as Bit'1 than what we expected to be Bit'0 but instead it inserted it as Bit'1. 
So in short:
For the bit field, the value from the text file is interpreted as "whether there is a value then it inserted as Bit'1 or there isn't any value (empty string), it inserted as Bit'0". Any value such as 0,1,2134,'True', 'foo', 'bla' is interpreted as Bit'1 and this is a surpise finding for me as I am currently working on migrating data from mssql to mysql. 
So to import Bit'0 from text file we need to set it to empty or none instead of e.g ,0
FYI: 
Field column with file path
For instance, we are importing an exported data .csv or .dat file from mssql server which one of the tables has a field named MyFilePathNameColumn and it stores a file's full path as c:\myfolder\myfile.txt in the .dat file so when we are importing this .dat file data to mysql using Load Data Infile method, we would expect it to be inserted as c:\myfolder\myfile.txt but instead it inserted it as c:myfoldermyfile.txt because mysql uses Linux's grammar in path convention so it uses forward slash in the file path like Linux hence back slash is removed.
To resolve this issue, when doing the query out of mssql data using bcp.exe we can use mssql's built in function REPLACE to replace '\' with '/' as seen:

select REPLACE(MyFilePathNameColumn, '\', '/')" from MyTable;

MSSQL look up any tables having specific Column Type in a database

E.g: To find any tables in a db having column type ‘uniqueidentifier‘ aka guid, we can execute following sql against the target db:

use MyDB;
SELECT SCHEMA_NAME(t.schema_id) AS schema_name,
t.name AS table_name,
c.name AS column_name,
c.max_length,
c.is_nullable,
c.is_computed
FROM sys.columns c
INNER JOIN sys.tables t ON t.object_id = c.object_id
WHERE
c.user_type_id IN (SELECT t.user_type_id FROM sys.types t WHERE t.name IN ('uniqueidentifier')) AND
t.is_ms_shipped = 0

E.g: To find any tables in a db having column type 'varbinary' aka blob or hex data, we can execute following sql against the target db:
use MyDB;
SELECT SCHEMA_NAME(t.schema_id) AS schema_name,
t.name AS table_name,
c.name AS column_name,
c.max_length,
c.is_nullable,
c.is_computed
FROM sys.columns c
INNER JOIN sys.tables t ON t.object_id = c.object_id
WHERE
c.user_type_id IN (SELECT t.user_type_id FROM sys.types t WHERE t.name IN ('varbinary')) AND
t.is_ms_shipped = 0

P.S.

To return list of column name, & column type from a table:

select COLUMN_NAME, DATA_TYPE from INFORMATION_SCHEMA.COLUMNS where TABLE_NAME = 'MyTable'

Python’s saga continue

Today I am learning reading file with python using with as : code block and some string manipulation. it is quite similiar to C# which I have spending many years coding with so I could have skipped this but I will finish learning it by the book.

Python’s saga continue

I have to agree that it is one of the powerful feature of this language. In C#, it has similiar feature but less power since you can only assign ‘same value’ to multiple variables within the same line. But with this powerful feature of python, you can assign multiple values to multiple variables within the same line. As of this writing, in C# it also now has this feature but still less powerful than this since we are required to use var keyword and install additional NuGet package called “System.ValueTuple”.

Anyways, I gave this powerful python’s feature a test and voila I like it:

Here is the practical example: we have a file name called fn which is assigned with string “Abi-10-50m-Back.txt” where Abi is the swimmer name, 10 is her age, 50m is her swimming distance and last but not least Back is her swimming style. Below example is to extract these information from the file name vairable fn into its own separate variable. I am happy with python as it can accomplish this task with mimal code than if using C# for the split string in C# items would be returned as an array. And to access each of them requires using arrary with its index and have to declare each of variable per arrary index for code readibility but with python, less code and it is easy to read.

Python’s saga continue

print dir combo mambo is very handy as other have said. We can see attributes/methods associated with any object in python. here now I can see all the methods I can use against my string object variable fn. And string in python is immutable which is the same thing in C#.

Now if I want to know; for instance, what the method zfill actually would do on the string object I can use print with help BIF which will print the documentation of zfill telling me all about zfill method’s fuctionality. (It might be equivalent C# String.padleft(…))

no se encontro la orden

My co-worker executed a script I wrote for automating a few thing from our day-to-day work so our work life is easier. My co-worker DM’ed me via Slack that there was an error and it returned as “no se encontro la orden”. I never seen an error like that in Linux before. My first thought that it must be French as it sounds like French but turned out it is Spanish after I google translated it. It means “Command not found” and once I understand that then it is easier to pinpoint what went wrong with the script in my co-worker’s environment. And we got it resolved.

I am bilingual myself and also knew a little Japanese since I learned it for 3 years while pursuing my Undergraduate degree in Computer Science. Also I am just recently interested in learning Spanish as well.

Linux switch network to DHCP or Static

I just helped a co-worker regarding this particular task where she wants to switch her network interface from manual static IP to automatic IP by dhcp in a linux client using command line since it does not have x desktop so I quickly wrote following script

you can find your interface name quickly by executing nmcli dev status at the terminal

How to Configure Network Connection Using 'nmcli' Tool

switchStaticToDHCP.sh

requires passing by one argument which is your network interface name.

!/bin/bash
 nmcli con mod $1 ipv4.method auto
 nmcli con mod $1 ipv4.gateway ""
 nmcli con mod $1 ipv4.address ""
 nmcli con down $1
 nmcli con up $1
 dhclient $1

switchStaticToDHCP.sh Your_Network_Interface_Name

eg: bash switchStaticToDHCP.sh enp0s3

switchDHCPToStatic.sh

requires passing by 3 arguments – your network interface name, static ip address, & default gateway ip address.

!/bin/bash
 nmcli con mod $1 ipv4.method manual
 nmcli con mod $1 ipv4.addresses $2/24
 nmcli con mod $1 ipv4.gateway $3
 nmcli con down $1
 nmcli con up $1

swtichDHCPToStatic.sh Your_Network_Interface_Name IP_Address Default_Gateway

eg: bash switchDHCPToStatic.sh enp0s3 192.168.1.168 192.168.1.254

Using self sign certificate with mono in Centos family Linux communicating with IIS Server

sometimes we do not want to pay for the Certificate authority – CA to issue a real certificate or for development, we just want to use a self sign cert that we signed and manually trusted it for testing https traffic just for development environment.

In Windows, we can just click on the self sign cert and it prompts us to where we want to store it which usually should be stored in Trusted Root Authority store.

Windows Cert. Trust - Trusted Root Certification Authorities vs Enterprise  Trust

But where is it in Linux and especially for mono? also I am not going to cover self sign cert for IIS server creation here in this post and I assume you already know how to do it. if you go to IIS manager (inetmgr) you should be able to create it there.

How to Install an SSL/TLS Certificate In Microsoft IIS 8 - The SSL Store™

I wrote a bash script here where it will download your IIS self signed cert from your IIS Server in pem format and then merge it with existing certificate bundle – tls-ca-bundle.pem. Then sync the newly merged cert list to mono store – Trust store

registerIISServerCertSelfSignForMono.sh

#!/bin/bash
echo "Server:Port = $1";

### Download IIS self sign cert in pem format from given server:port
openssl s_client -showcerts -verify 5 -connect $1 < /dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/) out="IISSelfSignCert.pem"; print >out}'

TEMPDIR=/etc/pki/ca-trust/source/temp

### Cleaning up our temp directory
rm -rf $TEMPDIR
mkdir -p $TEMPDIR

### copy our IIS Server self sign cert pem to our temp directory
cp IISSelfSignCert.pem $TEMPDIR

### Merge our IIS self sign cert pem to existing tls ca bundle cert pem
cp /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem $TEMPDIR/MergedCertList.pem

echo -e "\n# $f added for IIS Server Self Sign Cert" >> $TEMPDIR/MergedCertList.pem
cat $TEMPDIR/IISSelfSignCert.pem >> $TEMPDIR/MergedCertList.pem

cat $TEMPDIR/MergedCertList.pem

#now sync with Mono store, Trust store
cert-sync $TEMPDIR/MergedCertList.pem

Execute it at the terminal:

$ registerIISServerCertSelfSignForMono.sh yourServerName:Port

eg:

$ registerIISServerCertSelfSignForMono.sh www.vic-llc.org:443

What if you returns exit code -1 in bash, will bash take it as -1?

Actually no, if we return -1 in bash, it will be interpreted as 255 per https://tldp.org/LDP/abs/html/exitcodes.html

if I can go back in time, I would never return exit code -1 for a failure in Linux. 1 would be ideal in this case.

Let write a simple bash script test – to copy non-existent file

the execution will fail and exit status code is 1 as expected so let’s modify this script and return the exit status code to -1.

Since we alter our script by checking if exit code is not zero, we instead return -1. As a result, bash interprets it as out of range since exit takes only integer args in the range 0 – 255 and therefore interprets it as 255 instead of -1.