Scripting

Ramblings and links to interesting snippets


Site-wide search with Laravel Scout & Sphinx

I needed to set up a simple indexer to increase site performance.  After looking at the different options, like Elastic Search, Meilisearch, TNT Search, ect.  I ended up on Sphinx.  Sphinx is a little dated, but it’s very fast and doesn’t have a lot of dependencies, since it’s written in C++.

To start I installed Sphinx

1
sudo apt-get install sphinxsearch

Then updated the config file

1
2
cd /etc/sphinxsearch/
sudo nano sphinx.conf

Here’s a very simple full-text search example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
source database
{
  type          = mysql
 
  sql_host      = localhost
  sql_user      = user
  sql_pass      = secret
  sql_db        = database
  sql_port      = 3306 # optional, default is 3306
}
 
source src_articles : database
{
    sql_field_string = name
 
    sql_query   = SELECT id, name from articles
}
 
index articles
{
    type        = plain
    source      = src_articles
    path         = /var/lib/sphinxsearch/data/articles
 
    min_prefix_len = 3
 
    index_exact_words   = 1
    expand_keywords     = 1
 
    charset_type = utf-8
}
 
indexer
{
        mem_limit               = 256M
}
 
searchd
{
        listen                  = 9312
        listen                  = 9306:mysql41
 
        # log file, searchd run info is logged here
        # optional, default is 'searchd.log'
        log                     = /var/log/sphinxsearch/searchd.log
 
        # query log file, all search queries are logged here
        # optional, default is empty (do not log queries)
        query_log               = /var/log/sphinxsearch/query.log
 
        read_timeout        = 5
        max_children        = 30
        pid_file            = /var/run/sphinxsearch/searchd.pid
        seamless_rotate     = 1
        preopen_indexes     = 1
        unlink_old          = 1
        workers             = threads
        binlog_path         = /var/lib/sphinxsearch/data
        collation_server    = utf8_ci
        thread_stack        = 6144K
}

Then we need to start Sphinx, make sure to run it under the sphinxsearch user and run the index and create the cron to re-run the indexes once a day at midnight

1
2
3
4
5
6
7
8
sudo nano /etc/default/sphinxsearch and set START=yes
 
sudo su sphinxsearch -s /bin/bash
indexer --all
/etc/init.d/sphinxsearch start
 
sudo crontab -u sphinxsearch -e
0 0 * * * indexer --all --rotate

Then, install the plugins we need for Laravel

1
2
3
composer require laravel/scout:6.0.0
composer require constantable/laravel-scout-sphinx
php artisan vendor:publish --provider="Laravel\Scout\ScoutServiceProvider"

Then you need to add these into the providers in config/app.php

1
2
Constantable\SphinxScout\ServiceProvider::class,
Laravel\Scout\ScoutServiceProvider::class,

Alright! That should get Sphinx all set on the server, and ready to go in Laravel. In all models you want to be searchable, add this (also, they all need to have indexes defined in Sphinx settings)

1
2
3
use Laravel\Scout\Searchable;
 
use Searchable;

Now comes the controller method to intermingle the models :X This is still a WIP, scout has a very limited version of eloquent attached to it, so you can’t set the offset by hand. So this is as close as you can get with out-of-the-box Scout.

This supports a JSON version that intermingles 6 results from each source for an autocomplete field, as well as a paginated version.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
public function search(Request $request) {
        $rsx = $request->header('X-Requested-With');
 
        if (!($page = $request->get('page'))) {
            $page = 1;
        }
 
        $searchResults = [];
 
        if ($q = $request->get('q')) {
            $tables = ['article','video','book'];
 
            if ($rsx != 'XMLHttpRequest') {
                //get totals for pagination
                $total = 0;
                $totals = [];
                foreach ($tables as $k => $table) {
                    $totals[$k] = ('App\Models\\' . str_replace('_', '', ucwords($table, '_')))::search($q)->paginate(1);
                    $totals[$k] = $totals[$k]->total();
                    $total = $total + $totals[$k];
                }
 
                $pagination = new LengthAwarePaginator([], $total, $perPage);
                $pagination = str_replace('/?', '?q='.$q.'&', $pagination->render());
 
                $parts = sizeof($totals);
                foreach ($tables as $k => $table) {
                    $offset = ($page - 1) * 3;
                    if(($offset + 3) > $totals[$k]) {
                        $parts = $parts - 1;
                    }
                }
                if($parts==0)$parts=1;
                $perpart = 9 / $parts;
            }
 
            foreach($tables as $k => $table) {
                if ($rsx != 'XMLHttpRequest' && ($offset + 3) > $totals[$k] && $page != 1 && $totals[$k] != 0) {
                    continue;
                }
 
                $model = ('App\Models\\'.str_replace('_','',ucwords($table,'_')));
                $searchResults[$k] = $model::search($q);
                if ($rsx == 'XMLHttpRequest') {
                    $searchResults[$k] = $searchResults[$k]->take(6)->get()->all();
                } else {
                    $searchResults[$k] = $searchResults[$k]->paginate(floor($perpart)+($table == 'article' && is_float($perpart) ? 1 : 0))->all();
                }
 
                $searchResults[$k] = array_map(function ($i) use ($table) {$i->type = $table.'s';return $i;}, $searchResults[$k]);
            }
 
            $searchResults = $this->array_zip_merge($searchResults);
            $searchResults = array_filter($searchResults);
        }
 
        if ($rsx == 'XMLHttpRequest') {
            $rta = [];
            foreach ($searchResults as $s) {
                $rta[] = array(
                    'name' => $s['name'],
                    'date' => $s['updated_at'],
                    'link' => $s['link'],
                );
            }
            return json_encode($rta);
        }
 
        return view('search_results',['pagination'=>$pagination,'q'=>$q, 'searchResults'=>$searchResults]);
}
 
private function array_zip_merge($args) {
        $output = array();
        // The loop incrementer takes each array out of the loop as it gets emptied by array_shift().
        for ($args = $args; count($args); $args = array_filter($args)) {
            // &$arg allows array_shift() to change the original.
            foreach ($args as &$arg) {
                $output[] = array_shift($arg);
            }
        }
        return $output;
}

These resources were really helpful in getting everything set-up

https://laravel.com/docs/8.x/scout
https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-sphinx-on-ubuntu-14-04
https://sam-ngu.medium.com/site-wide-search-in-laravel-with-scout-27bbdc75ae8
https://github.com/franksierra/OSMServer

Permalink » No comments

Merge Excel Sheets with Laravel Excel

Laravel Excel allows you to create and format Excel documents with PHP within the Laravel framework.  I specifically needed to merge Excel sheets, and not just tack on the sheets to one document, but actually merge several files into a single sheet.  It’s pretty straightforward, but the documentation leaves a bit to be desired so it took me a while to narrow down the right way to do this.  The only downside is it will remove any formatting from the sheets.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
//first pull all the rows
$rows = [];
Excel::batch(storage_path('FOLDER/CONTAINING/EXCEL/FILES'), function($rows1, $file) use($rows) {
    $rows1->each(function($row) use($rows1) {
        global $rows;
 
        $rows[] = $row->toArray();
    });
 
});
 
//then add them to a new file
Excel::create('new_filename', function ($excel) use ($rows) {
    $excel->sheet('Sheet', function ($sheet) use ($rows) {
        global $rows;
 
        // Sheet manipulation
        foreach($rows as $row) {
            $sheet->appendRow($row);
        }
    });
})->store('xls', storage_path('excel/exports'));

Permalink » No comments

Compare CSV files in Bash

I’ve been needing this forever!  I have huge CSV lists and have needed a way to quickly compare them.  For this I just need to make sure that the first field from one CSV does not exist anywhere in the second CSV.  It was actually pretty simple to write this in Bash and it runs really fast!

1
2
3
4
5
6
while IFS=, read -r field1 field2
  do
    if grep --F "$field1" list1.csv; then
      echo "$field1 found"
    fi
done < list2.csv

Permalink » No comments

Engintron and mirrors

This is pretty much the same thing as what I did in a previous post, but with Engintron vs NGINX CP.  I had to do a lot of server updates to get PHP7 running, and the process broke NGINX CP, so I had to switch to Engintron.  NGINX by itself is much more efficient than Apache, and thus far has been plenty to keep the server running even when traffic spikes.  But for special occasions I do like to have a back-up plan in case the system gets overloaded so I have a mirror set and ready to go.

To first install Engintron you can follow these instructions:

  1. cd /
  2. rm -f engintron.sh
  3. wget –no-check-certificate https://raw.githubusercontent.com/engintron/engintron/master/engintron.sh
  4. bash engintron.sh install

One thing I found out is caching is not on for dynamic pages by default, in order to turn that on go to the Enigtron interface from WHM, then click on “Edit default.conf” find this code:

1
2
 set $CACHE_BYPASS_FOR_DYNAMIC 1;
 set $CACHE_BYPASS_FOR_STATIC 0;

and set it to:

1
2
 set $CACHE_BYPASS_FOR_DYNAMIC 0;
 set $CACHE_BYPASS_FOR_STATIC 0;

“bypass_for_dynamic” will essentially turn off caching for dynamic pages, if your dynamic pages aren’t being updated very often it’s best to have caching on.  If you have a specific page that cannot by cached you can turn the bypass back on using something like this:

1
2
3
 if ($SITE_URI ~* "yourdomain.com/url_to_not_cache") {
    set $CACHE_BYPASS_FOR_DYNAMIC 1;
 }

Like previously I’m using  the split clients module to spread out traffic to a mirror.  The logic here is a bit more tricky then in NGINX CP and I only have one mirror, so the $mirror variable is just determining whether or not I’ll be redirecting that user.  The code looks essentially like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
split_clients "${remote_addr}-{$query_string}" $mirror {
   5% "y";
   * "n";
}
 
server {
 
   if ($SITE_URI ~* "domain.com") {
      set $mirror "${mirror}maindomain";
   }
   if ($http_referer = "") {
      set $mirror "n";
   }
   if ($mirror = "ymaindomain") {
      return 302 "http://mirror.yourdomain.com/";
   }
 
}

So here the split_clients will set the $mirror variable to “y” 5% of the time.  Then, if the domain is “domain.com” it will set the $mirror variable to either “ymaindomain” or “nmaindomain” and the http_referer is just stopping all direct traffic from being redirected to the mirror.  The reason I’m changing the variable is so I don’t end up mirroring subdomains, and I could use that to have different mirrors for different subdomains or specific pages.

Permalink » No comments

How to Package cordova app

Just a simple step by step reference for packaging a cordova app:

  1. CD to the directory containing your app in the console
  2. export PATH=$PATH:”/cygdrive/P/Programs/Android/SDK/tools”
    export PATH=$PATH:”/cygdrive/P/Programs/Java/jdk1.8.0_121/bin”
    export PATH=$PATH:”/cygdrive/P/Programs/Android/SDK/build-tools/25.0.1″
    export ANDROID_HOME=/cygdrive/P/Programs/Android/SDK/
    export PATH=${PATH}:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools
    This is specific to my system, you may not need to do something like this
  3. cordova build –release android
    This will create platforms\android\build\outputs\apk\android-release-unsigned.apk
  4. CD and copy your keystore to platforms\android\build\outputs\apk
  5. jarsigner -verbose -sigalg SHA1withDSA -digestalg SHA1 -keystore your.keystore android-release-unsigned.apk keystorename
    sign it with your keystore, you may need to use a different signing method
  6. zipalign -v 4 android-release-unsigned.apk android-release-signed.apk

All done!

Permalink » No comments