Tuesday, December 27, 2016

Install GitLab Runner on openSUSE 42.1

It seems like openSUSE 42.1 isn't supported officially by GitLab Runner (or I didn't find a way)

But it is possible to use packages for other Linux distribution (RHEL 7 seems works)

so , next commands will install gitlab runner

>sudo zypper ar -t YUM https://packages.gitlab.com/runner/gitlab-ci-multi-runner/el/7/x86_64 runner_gitlab-ci-multi-runner
>sudo zypper --gpg-auto-import-keys refresh runner_gitlab-ci-multi-runner
>sudo zypper install gitlab-ci-multi-runner


then you can run

>sudo gitlab-ci-multi-runner register


and proceed with configuration according to docs https://docs.gitlab.com/runner/install/linux-repository.html





laravel-bootstrap-adminlte-starter-kit

Template for websites with basic functionality. It is based on next ideas:
  • have common features already integrated and configured (tests,gulp,bower etc)
  • simplify updates (via git merge from this project)
  • extensive use of .env config (slightly more then original Laravel)
  • 'make' based macro-tool for often used commands 
Home : https://gitlab.com/oprudkyi/laravel-bootstrap-adminlte-starter-kit/
Mirror: https://github.com/oprudkyi/laravel-bootstrap-adminlte-starter-kit 

Introduction

Just base functionality for other projects
Project includes already preconfigured resources:
Creation of new site based on starter kit
#clone original repository
git clone git@gitlab.com:oprudkyi/laravel-bootstrap-adminlte-starter-kit.git my_project

#cd to project
cd my_project

#delete origin
git remote rm origin

#use your new repository as main source 
git remote add origin git@gitlab.com:oprudkyi/web-site.git

#keep original source for updates
git remote add starter-kit git@gitlab.com:oprudkyi/laravel-bootstrap-adminlte-starter-kit.git

#push to your repository
git push -u origin master

 

Keeping your site in sync with starter kit

git fetch starter-kit
git merge starter-kit/master
git push
or
make merge-starterkit
git push

 

Installation

If you are building from the first time out of the source repository, you will need to generate the configure scripts. From the top directory, do:

./bootstrap.sh

Once the configure scripts are generated, 'make' system can be configured. From the top directory, do:

./configure

Run ./configure --help to see other configuration options
Install configured dependencies - tools like composer/bower and components as defined in composer.json, bower.json, package.json :

make download-dependencies

 

Testing

There are a large number of tests that can all be run from the top-level directory.
      make -k check
      make -k check-functional
      make -k check-acceptance
This will make all of the libraries (as necessary), and run through the unit tests defined in each of the client libraries. If a single language fails, the make check will continue on and provide a synopsis at the end.

 

Boostraping development environment

cp .env.example .env
vim .env

#for sqlite db
touch storage/database/play.sqlite
./artisan migrate

#generate key
./artisan key:generate -v

 

Update dependencies

make update-downloaded-dependencies

or

make update-downloaded-dependencies-production
 
it will search for updated packages for composer/npm/bower and install them as well will update lock/json files

 

Mailcatcher integration

Mailcatcher is configured in the .env.example for port 1025, tests use different port (11031) so tests can be run while development instance of maicatcher is running
Run:

make run-mailcatcher
 
Stop:

make stop-mailcatcher

Mailcatcher web gui available via http://127.0.0.1:1080/

 

Javascript/Css assets

Intergration is based on Laravel Elixir and Bower as package system (packages installed into vendor/bower_components).
To call bower use something like
node_modules/.bin/bower install "eonasdan-bootstrap-datetimepicker#latest" --save
or update current dependencies
node_modules/.bin/bower install
The building script is gulpfile.js. System is configured to generate single js file from all packages provided (as well single css file)
if you add/remove packages, you may also need to edit resources/assets/sass/app.scss as well for adding/removing css/scss/js
use 'make gulp' or 'make gulp-watch' to compile resources
custom css rules you can add to resources/assets/sass/starterkit/starter-kit-customize.scss it's applied after any other css, so it's possible to change any css behavior here

 

Manual Deployment Initialization

This isn't adivsed , just for info:
  • First, clone your repository to production/staging server
  • configure production environment
    ./bootstrap.sh
    ./configure --enable-tests=no
    make download-dependencies-production
    chmod 777 -R storage
  • create db (in case of mysql)
sudo mysql 
CREATE DATABASE IF NOT EXISTS starterkit_db CHARSET=utf8;
CREATE USER 'starterkit_user'@'%' IDENTIFIED BY 'starterkit_password';
GRANT ALL PRIVILEGES ON starterkit_db.* TO 'starterkit_user'@'%';
  • create env file
cp .env.example.production .env
vim .env
  • create key
php artisan key:generate
  • migrate db
php artisan migrate
  • optimize app
make production-optimize
it will run next commands:
#clear cache
php artisan view:clear
php artisan config:clear
php artisan route:clear
php artisan clear-compiled

#optimize
composer dump-autoload --optimize


#caching 
php artisan cache:clear
php artisan config:cache
php artisan route:cache
php artisan optimize

#recompile js/css
gulp

 

Manual Deployment

git pull
make download-dependencies-production
php artisan migrate
make production-optimize

Monday, December 19, 2016

Simple library to show JavaScript / jQuery errors as browser alerts

  • forgot to check browser's console log for errors ?
  • got complaints from users 'I click but nothing happens' ?
  • ever taught end-users how to look at browser's console ?
There is small and simple library to avoid or at least reduce possibility of such problems, specially in case if code isn't covered by tests.

It catches normally invisible JavaScript errors and exceptions and provides simple way to show them to developer/tester/user.

Simple errors are caught via window.onerror handler.
Errors inside jQuery callbacks (ajax, onclick etc) are hidden from window.onerror(), so jQuery is patched in way to call error handler.

Installation:

 
bower install js-error-alert --save
npm install js-error-alert

Load base script (it sets window.onerror):

<script src="window_error_handler.js"></script>

Add jQuery hook just after jQuery core code:

<script src="jquery-3.11.1.min.js"></script>
<script src="jquery_error_handler.js"></script>

Use custom handler to show errors:


You can create own JSEH_showUncaughtException(message) function and replace default one, place it at the top of the page (before window_error_handler.js)

var JSEH_showUncaughtException = function(message) {
    "use strict";

    if(typeof message === "undefined") {
        return false;
    }
    alert(message);
};

Enable/disable error handler:

In case you need to dynamically enable/disable handler (by example if merging/minifying tools are used) you can set JSEH_enabled variable to true/false.

var JSEH_enabled = false;  
//in Laravel/layout.blade.php 
var JSEH_enabled =@if(env('JS_ERROR_ALERTER_ENABLED', false)) true @else false @endif ;

Simple test

<script>
test_undefined_variable
</script>

<script>
$(function() {
    test_jquery_undefined_variable
});
</script>


Source : https://github.com/oprudkyi/js-error-alert

jQuery hooks are based on code found at :



Sunday, December 18, 2016

SOLVED. Laravel memcached issues. throttling stopped to work abruptly


After some use of Laravel's throttle feature (api/logins throttle) it started to return 'Too many attempts' when memcached cache is used (as well it is possible to have more issues, in case you depend on lifetime of cached data).
For me it happens after waking up PC from hibernation

short fix:
 - just restart memcached

possible explanation:
It seems like issue is related to the way how memcached uses expiration time of values and probably some bug in it. There are two modes possible - one, just number of seconds of how long memcached should keep value, second is Unix timestamp.

Laravel on other hand, uses latter approach (i.e. Unix timestamp) https://github.com/illuminate/cache/blob/master/MemcachedStore.php, function toTimestamp() 

while it works great all the time, after some sort of time de-synchronization (by example after suspending/hibernating of PC) Laravel might send expiration time in the past (because php/memcached are caching current timestamp) and memcached might (unchecked) just decide to keep item forever and break any functionality which depends on auto-deletion of values (like throttling 'lockout' key https://github.com/illuminate/cache/blob/master/RateLimiter.php )




Wednesday, September 21, 2016

concept of docker-compose-cauldron tool

The docker-compose.yml  format is very stiff (and Dockerfile as well ) :
  • environment variables (.env) don't allow to have default values, lists etc 
  • extend is very limited - multi-value sets are merged, links don't inherited etc
  • Dockerfile can have conditions for RUN, but not for other commands
  • there is no repository for compose.yml files and collections 
as the end every project ends with very specialized config which hard to support (updates, new features, fixes)

So there will be great to have tool which generates all needed files with use of some templating engine (if/for/while support inside docker-compose.yml, Dockerfile and other files as well)

use cases:

1. docker-compose-cauldron init {git url or name at repository}
- downloads source of template (into local subdir or somewhere under /home/.cache/
- creates docker-couldron.yml.example and docker-couldron.yml (project wide settings, i.e. will be stored in git) and .env.example (sample config for local environment (i.e. development/dev settings etc), stored in git but copied/edited to .env for local environment )

2. docker-compose-cauldron build
Builds docker-compose.yml and related "build" files for local environment (does not stored in git) based on settings from docker-couldron.yml and template

3. docker-compose-cauldron update
Updates source of template, as well docker-couldron.yml.example and .env.example , user should manually update docker-couldron.yml and .env files



Monday, September 5, 2016

SOLVED: docker-compose. Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

I configured new set of containers (via docker-compose),  and tried them to run

>docker-compose up -d
...
Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use 


googling returns me pretty inconvenient solutions:
  • just stop local web/mysql server or use different port for container. 
  • don't publish ports at all, i.e. remove '-ports'/'-p'  
  • use reverse-proxy before containers for web servers 
But I needed something less troublesome, I needed different/independent sets of containers running, i.e. different projects with different needs, with easy way to setup/remove, so I ended with next solution:
  • tie each set of containers (from single docker-compose.yml ) to docker custom networks i.e. all containers from set are bound to some unique IP range
  • use /etc/hosts to bind IPs to names  and always have regular ports 80, 443, 3306 etc accessible from local host
  • use of environment file (.env) for local config (so team can use the same config, while each member can customize important values) 

so, this is how it looks:
>cat .env
GATEWAY_IP=172.16.240.1
SUBNET=172.16.240.0/24
COMPOSE_PROJECT_NAME=myproject1

>cat docker-compose.yml
version: '2'

services:
  nginx:
    image: nginx:latest
    ports:
      - "${GATEWAY_IP}:80:80"
    networks:
      - export

networks:
  export:
    driver: bridge
    ipam:
      driver: default
      config:
      - subnet: "${SUBNET}"
        gateway: "${GATEWAY_IP}"

and docker merges both files and has config as:
>docker-compose config                  
networks:
  export:
    driver: bridge
    ipam:
      config:
      - gateway: 172.16.240.1
        subnet: 172.16.240.0/24
      driver: default
services:
  nginx:
    image: nginx:latest
    networks:
      export: null
    ports:
    - 172.16.240.1:80:80
version: '2.0'
volumes: {}


>docker-compose up -d                   
Creating network "myproject1_export" with driver "bridge"
Creating myproject1_nginx_1


>docker-compose ps                    
       Name                Command          State                Ports              
------------------------------------------------------------------------------------
myproject1_nginx_1   nginx -g daemon off;   Up      443/tcp, 172.16.240.1:80->80/tcp



Now, just add into your /etc/hosts file some like
172.16.240.1 myprojecthost

and go to http://myprojecthost to check if it works (MySQL will be on myprojecthost:3306 etc)

Some additional notes:
  • you will need to link every service in docker-compose.yml  to same network, i.e.

        networks:
          - export


    Otherwise, docker-compose will create two networks - default and custom one, and if services don't share common network, they won't communicate.
    Of course, you can use default network too for services you want to publish:

        networks:
          - export
         
          - default

    then there no need to update every service
     
  • do not add .env in git , just create .env.sample , so every developer can copy it to .env and use customized config 
  • docker-compose names containers based on directory name, this is inconvenient if you work with different project with similar directory structure, because docker tools checks only name of container (i.e. down/restart etc will catch containers from other directories).
    To make things manageable COMPOSE_PROJECT_NAME environment variable is used. Still, it should be unique per docker host/machine, or docker tools will work incorrectly