0%

npm analyzer project: part 4

Posted on Edited on Disqus:

Install elasticsearch

elasticsearch is a scalable and speedy search, analytics, and storage.

import signing key (PGP key)

1
2
vagrant@cli:~/npm_analyzer$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK

install from apt repo

install the apt-transport-https package on Debian before proceeding.

1
2
3
4
5
6
vagrant@cli:~/npm_analyzer$ sudo apt-get install apt-transport-https
Reading package lists... Done
Building dependency tree       
Reading state information... Done
apt-transport-https is already the newest version (1.8.2).
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.

Save the repository definition to /etc/apt/sources.list.d/elastic-7.x.list

1
2
vagrant@cli:~/npm_analyzer$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
deb https://artifacts.elastic.co/packages/7.x/apt stable main

check source list

1
2
3
4
5
6
7
8
vagrant@cli:/etc/apt/sources.list.d$ ls -al
total 24
drwxr-xr-x 2 root root 4096 Nov  6 18:05 .
drwxr-xr-x 7 root root 4096 Nov  6 18:02 ..
-rw-r--r-- 1 root root  419 Nov  2 06:19 bintray.rabbitmq.list
-rw-r--r-- 1 root root   62 Nov  6 18:05 elastic-7.x.list
-rw-r--r-- 1 root root  189 Oct 17 04:04 google-chrome.list
-rw-r--r-- 1 root root   55 Oct 17 04:01 google-chrome.list.save

install the Elasticsearch Debian package

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
vagrant@cli:~/npm_analyzer$ sudo apt-get update && sudo apt-get install elasticsearch
Get:1 http://security.debian.org/debian-security buster/updates InRelease [39.1 kB]
Hit:2 http://deb.debian.org/debian buster InRelease                                                                                          
Get:3 https://download.docker.com/linux/debian buster InRelease [44.4 kB]                                                                    
Get:4 https://artifacts.elastic.co/packages/7.x/apt stable InRelease [7,124 B]                                                               
Ign:5 http://dl.google.com/linux/chrome/deb stable InRelease                                                                                 
Hit:6 https://deb.nodesource.com/node_10.x buster InRelease                                                                                  
Hit:7 http://dl.google.com/linux/chrome/deb stable Release                                                                                   
Ign:8 https://dl.bintray.com/rabbitmq-erlang/debian bionic InRelease                                              
Get:9 http://ftp.de.debian.org/debian buster-backports InRelease [46.7 kB]
Ign:10 https://dl.bintray.com/rabbitmq/debian bionic InRelease
Get:11 https://dl.bintray.com/rabbitmq-erlang/debian bionic Release [12.6 kB]
Get:12 https://dl.bintray.com/rabbitmq/debian bionic Release [74.5 kB]
Get:13 http://security.debian.org/debian-security buster/updates/main Sources [84.1 kB]
Get:14 http://security.debian.org/debian-security buster/updates/main amd64 Packages [112 kB]
Get:15 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 Packages [21.9 kB]
Get:19 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages.diff/Index [27.8 kB]                                             
Get:20 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-02-1415.02.pdiff [178 B]                                 
Get:21 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-03-0215.42.pdiff [276 B]                                 
Get:22 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-03-0818.43.pdiff [15.7 kB]                               
Get:23 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-03-2017.32.pdiff [501 B]                                 
Get:24 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-05-1414.24.pdiff [5,640 B]                               
Get:25 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-06-1414.27.pdiff [237 B]                                 
Get:25 http://ftp.de.debian.org/debian buster-backports/main amd64 Packages 2019-11-06-1414.27.pdiff [237 B]                                 
Fetched 493 kB in 7s (72.2 kB/s)                                                                                                             
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  elasticsearch
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 289 MB of archives.
After this operation, 488 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 elasticsearch amd64 7.4.2 [289 MB]
Fetched 289 MB in 29s (10.0 MB/s)                                                                                                            
Selecting previously unselected package elasticsearch.
(Reading database ... 57026 files and directories currently installed.)
Preparing to unpack .../elasticsearch_7.4.2_amd64.deb ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Unpacking elasticsearch (7.4.2) ...
Setting up elasticsearch (7.4.2) ...
Created elasticsearch keystore in /etc/elasticsearch
Processing triggers for systemd (241-7~deb10u1) ...

running elasticsearch with systemd

1
2
3
4
5
vagrant@cli:~/npm_analyzer$ sudo /bin/systemctl daemon-reload
vagrant@cli:~/npm_analyzer$ sudo /bin/systemctl enable elasticsearch.service
Synchronizing state of elasticsearch.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable elasticsearch
Created symlink /etc/systemd/system/multi-user.target.wants/elasticsearch.service → /lib/systemd/system/elasticsearch.service.

By default the Elasticsearch service doesn’t log information in the systemd journal. To enable journalctl logging, the –quiet option must be removed from the ExecStart command line in the elasticsearch.service file.

1
2
vagrant@cli:/lib/systemd/system$ vim elasticsearch.service
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

delete the “–quiet”

1
ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid

Note: do not forget to deamon-reload and restart elasticserch.service

check the journal

1
2
3
vagrant@cli:/lib/systemd/system$ sudo journalctl --unit elasticsearch
-- Logs begin at Tue 2019-11-05 07:29:08 GMT, end at Wed 2019-11-06 18:25:17 GMT. --
-- No entries --

check elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
root@cli:/home/vagrant/npm_analyzer# curl localhost:9200
{
  "name" : "cli",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "S9dLNZ_rTpa30PX5UT3VJg",
  "version" : {
    "number" : "7.4.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25e96",
    "build_date" : "2019-10-28T20:40:44.881551Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Add these configurations to the elasticsearch.yml

1
root@cli:/etc/elasticsearch# echo "action.auto_create_index: -npms-current,-npms-new,+*"  >> elasticsearch.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
root@cli:/etc/elasticsearch# ls
elasticsearch.keystore	elasticsearch.yml  jvm.options	log4j2.properties  role_mapping.yml  roles.yml	users  users_roles
root@cli:/etc/elasticsearch# cat elasticsearch.yml 
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
action.auto_create_index: -npms-current,-npms-new,+*

Install the head to perform various manual operations in a web GUI

elasticsearch-head is a web front end for an Elasticsearch cluster

1
2
3
4
5
6
7
8
root@cli:/home/vagrant/npm_analyzer# git clone git://github.com/mobz/elasticsearch-head.git
Cloning into 'elasticsearch-head'...
remote: Enumerating objects: 77, done.
remote: Counting objects: 100% (77/77), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 4337 (delta 38), reused 45 (delta 17), pack-reused 4260
Receiving objects: 100% (4337/4337), 2.51 MiB | 3.80 MiB/s, done.
Resolving deltas: 100% (2411/2411), done.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
root@cli:/home/vagrant/npm_analyzer# cd elasticsearch-head/
root@cli:/home/vagrant/npm_analyzer/elasticsearch-head# npm install
npm WARN deprecated coffee-script@1.10.0: CoffeeScript on NPM has moved to "coffeescript" (no hyphen)
npm WARN deprecated http2@3.3.7: Use the built-in module in node 9.0.0 or newer, instead
npm WARN deprecated phantomjs-prebuilt@2.1.16: this package is now deprecated
npm WARN deprecated json3@3.3.2: Please use the native JSON object instead of JSON 3
npm WARN deprecated json3@3.2.6: Please use the native JSON object instead of JSON 3

> phantomjs-prebuilt@2.1.16 install /home/vagrant/npm_analyzer/elasticsearch-head/node_modules/phantomjs-prebuilt
> node install.js

PhantomJS not found on PATH
Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
Saving to /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
Receiving...
  [======================================--] 96%
Received 22866K total.
Extracting tar contents (via spawned process)
Removing /home/vagrant/npm_analyzer/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom
Copying extracted folder /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2-extract-1573068758515/phantomjs-2.1.1-linux-x86_64 -> /home/vagrant/npm_analyzer/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom
Writing location.js file
Done. Phantomjs binary available at /home/vagrant/npm_analyzer/elasticsearch-head/node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs

> core-js@2.6.10 postinstall /home/vagrant/npm_analyzer/elasticsearch-head/node_modules/core-js
> node postinstall || echo "ignore"

Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!

The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
> https://opencollective.com/core-js 
> https://www.patreon.com/zloirock 

Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN elasticsearch-head@0.0.0 license should be a valid SPDX license expression
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

added 512 packages from 620 contributors and audited 1768 packages in 115.069s
found 40 vulnerabilities (19 low, 2 moderate, 19 high)
  run `npm audit fix` to fix them, or `npm audit` for details


   ╭────────────────────────────────────────────────────────────────╮
   │                                                                │
   │       New minor version of npm available! 6.9.0 → 6.12.1       │
   │   Changelog: https://github.com/npm/cli/releases/tag/v6.12.1   │
   │               Run npm install -g npm to update!                │
   │                                                                │
   ╰────────────────────────────────────────────────────────────────╯

start a local webserver running on port 9100 serving elasticsearch-head

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
root@cli:/home/vagrant/npm_analyzer/elasticsearch-head# curl localhost:9100
<!DOCTYPE html>

<html>
	<head>
		<meta charset="UTF-8">
		<title>elasticsearch-head</title>
		<link rel="stylesheet" href="_site/base/reset.css">
		<link rel="stylesheet" href="_site/vendor.css">
		<link rel="stylesheet" href="_site/app.css">
		<script src="_site/i18n.js" data-baseDir="_site/lang" data-langs="en,fr,pt,zh,zh-TW,tr,ja"></script>
		<script src="_site/vendor.js"></script>
		<script src="_site/app.js"></script>
		<script>
			window.onload = function() {
				if(location.href.contains("/_plugin/")) {
					var base_uri = location.href.replace(/_plugin\/.*/, '');
				}
				var args = location.search.substring(1).split("&").reduce(function(r, p) {
					r[decodeURIComponent(p.split("=")[0])] = decodeURIComponent(p.split("=")[1]); return r;
				}, {});
				new app.App("body", {
					id: "es",
					base_uri: args["base_uri"] || base_uri,
					auth_user : args["auth_user"] || "",
					auth_password : args["auth_password"],
					dashboard: args["dashboard"]
				});
			};
		</script>
		<link rel="icon" href="_site/base/favicon.png" type="image/png">
	</head>
	<body></body>
</html>

Note: In order to enable CORS in elasticsearch, so I add these commands to /etc/elasticsearch/elasticsearch.yml

1
2
http.cors.enabled: true
http.cors.allow-origin: "*"

also need to modify the network host in elasticsearch.yml

1
2
network.host: 0.0.0.0
discovery.seed_hosts: ["0.0.0.0", "127.0.0.1"]

then restart the elasticsearch

go to localhost:9100, you can connect to elasticsearch