Tuesday, April 7, 2015

add openssl modules to perl on windows

problem

Some perl cpan modules required openssl installed for your pc. Here are some of them:
  • Crypt::OpenSSL::AES
  • Crypt::OpenSSL::Bignum
  • Crypt::OpenSSL::Bignum::CTX
  • Crypt::OpenSSL::Blowfish
  • Crypt::OpenSSL::Blowfish::CFB64
  • Crypt::OpenSSL::CA
  • Crypt::OpenSSL::CA::AlphabetSoup
  • Crypt::OpenSSL::CA::Inline::C
  • Crypt::OpenSSL::CA::Resources
  • Crypt::OpenSSL::Cloner
  • Crypt::OpenSSL::Common
  • Crypt::OpenSSL::DSA
  • Crypt::OpenSSL::DSA::Signature
  • Crypt::OpenSSL::EC
  • Crypt::OpenSSL::ECDSA
  • Crypt::OpenSSL::PBKDF2
  • Crypt::OpenSSL::PKCS10
  • Crypt::OpenSSL::PKCS12
  • Crypt::OpenSSL::Random
  • Crypt::OpenSSL::RC4
  • Crypt::OpenSSL::RSA
  • Crypt::OpenSSL::SMIME
  • Crypt::OpenSSL::VerifyX509
  • Crypt::OpenSSL::X509
  • Crypt::SSLeay
That happens bcs they are just warppers over openssl functions.
When you are trying install such module - you get source code, which includes some header from openssl library & trying link with openssl library on your pc.
on unix-like systems this problem seems solves quickly - any of next commands:
  • sudo yum install perl-Crypt-SSLeay
  • sudo yum install openssl-devel
  • sudo apt-get install libssl-dev
that's so simple on unix-like systems because in unix you have some standard paths to headers & libraries. So, when you need to find library - you know where to search. On Windows, users don't have such cool thing - that's source of this problem.
And I didn't find any solutions of such problem in the Internet - so, solved & writed it here. Also I'm not pretend on some knowledge perl or it's package managers or something like that - I'm just using perl for solving my tasks, encounted problem and solved it.
In this note I will describe common principles of solving such problems - so, I think this experience can be expanded to any such problem.
to pass lyrics and just get solution - just scroll to "problem solving in a short".


tools & versions

I will describe on example of ActivePerl (and 64 and 32 bit version). Don't know what about strawberry perl, for example - but I think common idea will be the same.
I have ActivePerl v5.18.4
  • installed in C:\Perl64 for x86-64 version.
  • installed in C:\Perl for x86 version.
(I didn't install both of them in time - it must lead to confusion, but I've checked this approach on both versions - so, when you will see action for two versions - x86 and x86-64 - just choose your one)

Also you need to build openssl with mingw (for x86-64 target platforms - with mingw64).
How to do that - I described in note earlier - [link].
Why exactly mingw - bcs it's much simpler than add VisualStudio-compiled openssl. In the end of this note I will say how to add openssl, compiled in Visual Studio - but I don't like that.


Crypt::SSLeay

Module Crypt::SSLeay has easier solution that other mentioned, bcs it has some extra opportunities for installing. It has way to know where this libraries located ([link]):
> set OPENSSL_INCLUDE=...
> set OPENSSL_LIB=...
> cpanm Crypt::SSLeay

So, if you need only "Crypt::SSLeay" - you can just use it.
But this is specific only for this module - so, it's not universal method.


lyrics

What happened when you are trying install such module - for example 'cpan install Crypt::OpenSSL::Blowfish'? Well, first of all - if it's your fist 'cpan install' when you need compile stuff - it will bring you tools for compiling code (with mingw). You will see something like that:
--------------------------------------------------------------------------------------------------------
It looks like you don't have a C compiler and make utility installed.  Trying
to install dmake and the MinGW gcc compiler using the Perl Package Manager.
This may take a a few minutes...

Syncing site PPM database with .packlists...done
Downloading MinGW-4.6.3...done
Downloading dmake-4.11.20080107...done
Unpacking MinGW-4.6.3...done
Unpacking dmake-4.11.20080107...done
Generating HTML for MinGW-4.6.3...done
Generating HTML for dmake-4.11.20080107...done
Updating files in site area...
3697 files installed

--------------------------------------------------------------------------------------------------------
This utils will be located here:
    C:\Perl\cpan\build
or
    C:\Perl64\cpan\build

Then it will create directory in your cpan build directory
    C:\Perl\cpan\build
or
    C:\Perl64\cpan\build
in format:
    %package_name%-%version%-%six_looked_random_symbols%
(also it put .yml file with the same name, but for describing task it doesn't matter)

and here are extracted package archive and in this directory will be building process.
for example in 'Crypt-OpenSSL-RSA-0.28-gB_yPv'

here are located Makefile.PL file. For Crypt::OpenSSL this script uses module ExtUtils::MakeMaker. This 'Makefile.PL' script generates 'Makefile'. And here are the key thing - this script searches libraries and generates make file to linking with libraries, present on your pc!
When it can't find library it prints to err console stream messages like that:
...
Warning (mostly harmless): No library found for -lcrypto
Warning (mostly harmless): No library found for -lmoldname
...


Then, after Makefile.PL generated makefile, called 'dmake' util, which will try compile stuff accordingly to information in 'makefile' (on example installing Crypt::OpenSSL::Blowfish)
Blowfish.xs:5:30: fatal error: openssl/blowfish.h: No such file or directory
or stuff like that


So, the first step to solve this problem - copy your openssl includes directory to place where perl will find it:
    C:\Perl\site\lib\auto\MinGW\i686-w64-mingw32\include
or
    C:\Perl64\site\lib\auto\MinGW\x86_64-w64-mingw32\include

Then if you will try again 'cpan install Crypt::OpenSSL::Blowfish', it will find includes, but not find libraries:
Warning (mostly harmless): No library found for -lcrypto
...
Blowfish.o:Blowfish.c:(.text+0x125): undefined reference to `BF_encrypt'
Blowfish.o:Blowfish.c:(.text+0x151): undefined reference to `BF_decrypt'
Blowfish.o:Blowfish.c:(.text+0x2c3): undefined reference to `BF_set_key'


so, you need to put libraries where ExtUtils::MakeMaker will find it. Where ExtUtils::MakeMaker searches this files - you can understand by 'Process Monitor' tool [link]. Just turn on logging all events, run 'cpan install Crypt::OpenSSL::Blowfish', wait error message, save log to csv file and then grep by that. So, finally you will get list of locations where it searches libraries (part of log, grepped by 'ssl'):
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\-lssl32"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\libssl32.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\libssl32.a"
        "C:\Perl64\lib\CORE\libssl32.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\libssl32.dll.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\libssl32.dll.a"
        "C:\Perl64\lib\CORE\libssl32.dll.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\libssl32.lib"
        "C:\MinGW\x86_64-w64-mingw32\lib\libssl32.lib"
        "C:\Perl64\lib\CORE\libssl32.lib"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\ssl32.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\ssl32.a"
        "C:\Perl64\lib\CORE\ssl32.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\ssl32.dll.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\ssl32.dll.a"
        "C:\Perl64\lib\CORE\ssl32.dll.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\ssl32.lib"
as you can see - it searches libraries by names, which differs from that we got after compilcation. And also you can find "C:\Perl64\lib\CORE" ("C:\Perl64\lib\CORE" for x86-64) the best place to putting your libraries. I put it on these paths with these names:
    libssl.a    =>    C:\Perl64\lib\CORE\ssl32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\eay32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\libcrypto.a
+ copy this file (somewhy it searcher stuff like GetDeviceCaps):
    C:\Perl64\lib\CORE\libgdi32.a => C:\Perl64\lib\CORE\libgdi32.a
that's it.


dmake case

What if you have similar case, but it's more difficult?
For example - you have some dir with files in your C:\Perl\cpan\build or C:\Perl64\cpan\build directory - which stay here after unsuccessful installation of some cpan module. And you found way to fix makefile to build process finished correctly.
Then, to finish installation exec next commands:
> dmake
> dmake install



visual studio

Here are way to build with VisualStudio-compiled openssl.
First, when you put your lib files and try install some module, you will get:
        C:\Perl64\lib\CORE\libcrypto.a(tmp32/bf_cfb64.obj):(.text$mn+0x1e): undefined reference to `__chkstk'
        C:\Perl64\lib\CORE\libcrypto.a(tmp32/bf_skey.obj):(.text$mn+0x11): undefined reference to `__chkstk'
        C:\Perl64\lib\CORE\libcrypto.a(tmp32/bf_enc.obj):(.text$mn+0x1a): undefined reference to `__chkstk'
so, you need take
    C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\lib\chkstk.obj
for x86 or
    C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\lib\amd64\chkstk.obj
for x86-64.
put it somewhere in accessible place, for example
    C:\Perl64\lib\CORE\chkstk.a
and fix in makefile (in 'C:\Perl64\cpan\build'):

    LDLOADLIBS = C:\Perl64\lib\CORE\libcrypto.a C:\Perl64\lib\CORE\libgdi32.a
to
    LDLOADLIBS = C:\Perl64\lib\CORE\libcrypto.a C:\Perl64\lib\CORE\libgdi32.a C:\Perl64\lib\CORE\chkstk.a

then
> dmake
> dmake install
That's it. But I think way with mingw-compiled library is much simpler.


problem solving in a short

1. put includes from your mingw-compiled lib to:

    C:\Perl\site\lib\auto\MinGW\i686-w64-mingw32\include
or
    C:\Perl64\site\lib\auto\MinGW\x86_64-w64-mingw32\include

2. put your mingw-compiled lib to
    libssl.a    =>    C:\Perl64\lib\CORE\ssl32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\eay32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\libcrypto.a
    + copy
    C:\Perl64\lib\CORE\libgdi32.a => C:\Perl64\lib\CORE\libgdi32.a


useful sources & links
here are looks like another solution (but as I see - it's for strawberry perl) - [link]


APPENDIX - possible errors (for googling people):

main problem illustration - on example of 'cpan install Crypt::OpenSSL::SMIME'
------------------------------------------------------------------------------------------------------
Warning (mostly harmless): No library found for -lcrypto
Warning (mostly harmless): No library found for -lmoldname
Warning (mostly harmless): No library found for -lkernel32
Warning (mostly harmless): No library found for -luser32
Warning (mostly harmless): No library found for -lgdi32
Warning (mostly harmless): No library found for -lwinspool
Warning (mostly harmless): No library found for -lcomdlg32
Warning (mostly harmless): No library found for -ladvapi32
Warning (mostly harmless): No library found for -lshell32
Warning (mostly harmless): No library found for -lole32
Warning (mostly harmless): No library found for -loleaut32
Warning (mostly harmless): No library found for -lnetapi32
Warning (mostly harmless): No library found for -luuid
Warning (mostly harmless): No library found for -lws2_32
Warning (mostly harmless): No library found for -lmpr
Warning (mostly harmless): No library found for -lwinmm
Warning (mostly harmless): No library found for -lversion
Warning (mostly harmless): No library found for -lodbc32
Warning (mostly harmless): No library found for -lodbccp32
Warning (mostly harmless): No library found for -lcomctl32
Please specify prototyping behavior for SMIME.xs (see perlxs manual)
SMIME.xs: In function 'XS_Crypt__OpenSSL__SMIME_new':
SMIME.xs:232:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:233:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:234:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:235:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:236:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:237:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:238:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:245:18: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]
SMIME.o:SMIME.c:(.text+0x1a1): undefined reference to `BIO_new_file'
SMIME.o:SMIME.c:(.text+0x514): undefined reference to `X509_free'
SMIME.o:SMIME.c:(.text+0x520): undefined reference to `EVP_PKEY_free'
SMIME.o:SMIME.c:(.text+0x529): undefined reference to `X509_STORE_free'
SMIME.o:SMIME.c:(.text+0x535): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x61c): undefined reference to `X509_STORE_new'
SMIME.o:SMIME.c:(.text+0x62d): undefined reference to `X509_LOOKUP_file'
SMIME.o:SMIME.c:(.text+0x638): undefined reference to `X509_STORE_add_lookup'
SMIME.o:SMIME.c:(.text+0x660): undefined reference to `X509_LOOKUP_ctrl'
SMIME.o:SMIME.c:(.text+0x669): undefined reference to `X509_LOOKUP_hash_dir'
SMIME.o:SMIME.c:(.text+0x674): undefined reference to `X509_STORE_add_lookup'
SMIME.o:SMIME.c:(.text+0x693): undefined reference to `X509_LOOKUP_ctrl'
SMIME.o:SMIME.c:(.text+0x698): undefined reference to `ERR_clear_error'
SMIME.o:SMIME.c:(.text+0x6e7): undefined reference to `X509_STORE_free'
SMIME.o:SMIME.c:(.text+0x72e): undefined reference to `BIO_s_file'
SMIME.o:SMIME.c:(.text+0x736): undefined reference to `BIO_new'
SMIME.o:SMIME.c:(.text+0x754): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0x7b4): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x7d6): undefined reference to `d2i_PrivateKey_bio'
SMIME.o:SMIME.c:(.text+0x876): undefined reference to `d2i_PKCS12_bio'
SMIME.o:SMIME.c:(.text+0x895): undefined reference to `PKCS12_parse'
SMIME.o:SMIME.c:(.text+0x89d): undefined reference to `PKCS12_free'
SMIME.o:SMIME.c:(.text+0x905): undefined reference to `PEM_read_bio_PrivateKey'
SMIME.o:SMIME.c:(.text+0x938): undefined reference to `BIO_s_file'
SMIME.o:SMIME.c:(.text+0x940): undefined reference to `BIO_new'
SMIME.o:SMIME.c:(.text+0x967): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0x9d2): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x9ea): undefined reference to `d2i_PKCS12_bio'
SMIME.o:SMIME.c:(.text+0xa08): undefined reference to `PKCS12_parse'
SMIME.o:SMIME.c:(.text+0xa10): undefined reference to `PKCS12_free'
SMIME.o:SMIME.c:(.text+0xa4d): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xa6c): undefined reference to `PEM_read_bio_X509_AUX'
SMIME.o:SMIME.c:(.text+0xd7b): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xd9c): undefined reference to `PKCS7_sign'
SMIME.o:SMIME.c:(.text+0xdba): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xdd5): undefined reference to `BIO_new_file'
SMIME.o:SMIME.c:(.text+0xdf7): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xe0d): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xe23): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xe41): undefined reference to `SMIME_write_PKCS7'
SMIME.o:SMIME.c:(.text+0xe4d): undefined reference to `PKCS7_free'
SMIME.o:SMIME.c:(.text+0xe52): undefined reference to `sk_new_null'
SMIME.o:SMIME.c:(.text+0xe70): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xe9e): undefined reference to `sk_push'
SMIME.o:SMIME.c:(.text+0xeb2): undefined reference to `BIO_new_file'
SMIME.o:SMIME.c:(.text+0xee1): undefined reference to `PKCS7_encrypt'
SMIME.o:SMIME.c:(.text+0xeff): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xf15): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xf2b): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xf41): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xf5f): undefined reference to `SMIME_write_PKCS7'
SMIME.o:SMIME.c:(.text+0xf6b): undefined reference to `PKCS7_free'
SMIME.o:SMIME.c:(.text+0xf79): undefined reference to `X509_free'
SMIME.o:SMIME.c:(.text+0xf7e): undefined reference to `sk_pop_free'
SMIME.o:SMIME.c:(.text+0xf87): undefined reference to `X509_free'
SMIME.o:SMIME.c:(.text+0xf93): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0xf9f): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x1678): undefined reference to `OPENSSL_add_all_algorithms_noconf'
SMIME.o:SMIME.c:(.text+0x167d): undefined reference to `EVP_des_ede3_cbc'
c:/perl64/site/lib/auto/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.6.3/../../../../x86_64-w64-mingw32/bin/ld.exe: SMIME.o: bad reloc address 0x0 in section `.pdata'
c:/perl64/site/lib/auto/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.6.3/../../../../x86_64-w64-mingw32/bin/ld.exe: final link failed: Invalid operation
collect2: ld returned 1 exit status
dmake.exe:  Error code 129, while making 'blib\arch\auto\Crypt\OpenSSL\SMIME\SMIME.dll'
------------------------------------------------------------------------------------------------------

what to do: make actions mentioned in this note.

problem, when includes found, and library not found.
------------------------------------------------------------------------------------------------------
CPAN: Module::CoreList loaded ok (v5.20141002)
cp Blowfish.pm blib\lib\Crypt\OpenSSL\Blowfish.pm
Running Mkbootstrap for Crypt::OpenSSL::Blowfish ()
C:\Perl64\bin\perl.exe -MExtUtils::Command -e chmod -- 644 Blowfish.bs
C:\Perl64\bin\perl.exe C:\Perl64\lib\ExtUtils\xsubpp  -typemap C:\Perl64\lib\ExtUtils\typemap  Blowfish.xs > Blowfis
h.xsc && C:\Perl64\bin\perl.exe -MExtUtils::Command -e mv -- Blowfish.xsc Blowfish.c
C:\Perl64\site\bin\gcc.exe -c           -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUST
OMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fno-strict-aliasing -mms-bitfields -s -O2         -D
VERSION=\"0.02\"        -DXS_VERSION=\"0.02\"  "-IC:\Perl64\lib\CORE"  -DPERL5 -DOPENSSL_NO_KRB5 Blowfish.c
Blowfish.xs:5:30: fatal error: openssl/blowfish.h: No such file or directory
compilation terminated.
dmake.exe:  Error code 129, while making 'Blowfish.o'
  VKRAMSKIH/Crypt-OpenSSL-Blowfish-0.02.tar.gz
  C:\Perl64\site\bin\dmake.exe -- NOT OK
CPAN: YAML::XS loaded ok (v0.52)
Running make test
  Can't test without successful make
Running make install
  Make had returned bad status, install seems impossible
------------------------------------------------------------------------------------------------------

what to do: make actions mentioned in this note - after putting includes in accessible place.

1 comment: