Installing and using Solr 6.1 cloud in AWS EC2 (with some note on upgrading)

Like any company, we also have some legacy codes. Our codes were using Solr 3 and I was going to upgrade it to the latest (6.1). The upgrading itself is not such a big deal, just fire up a new setup convert the old schema type to the new schema type which only differs in XML formats. I am not going through that as you can easily get sample schema format from latest version and just compare it to your schema. Once done you can start the new solr with your old schema and it will start giving errors!! but with patience and hard work you can resolve them one by one.

Anyway, the upgrade process is not such a big deal but working with new solr is. Specially if you want to use the cloud version which uses zookeeper to manage the configs, shards, replications, leaders and etc. All you might come on your way is some depreciated class or missing class which you can download.

In my case I found this page very useful to find the deprecated classes of Solr 3.6.

Before I jump on Solr cloud 6.1 you may need to know some concepts:

  1. Collection: A single search index.
  2. Shard: A logical section of a single collection (also called Slice). Sometimes people will talk about “Shard” in a physical sense (a manifestation of a logical shard). Shard is literally the parts of your data. It means if you have 3 shards then all your data (documents) are distributed in 3 parts. It also means if one of the shards is missing then you are in trouble!!
  3. Replica: A physical manifestation of a logical Shard, implemented as a single Lucene index on a SolrCore. Replica is the replication of the shards! so if you have replication factor of 2 then you will have 2 copy of each shard.
  4. Leader: One Replica of every Shard will be designated as a Leader to coordinate indexing for that Shard. Leader is the master node in a shard. So if you have to replicas, then the master one is the boss!
  5. SolrCore: Encapsulates a single physical index. One or more make up logical shards (or slices) which make up a collection.
  6. Node: A single instance of Solr. A single Solr instance can have multiple SolrCores that can be part of any number of collections.
  7. Cluster: All of the nodes you are using to host SolrCores.

In continue, I will go through installing and using this whole setup.

Continue reading “Installing and using Solr 6.1 cloud in AWS EC2 (with some note on upgrading)”

Installing and using Solr 6.1 cloud in AWS EC2 (with some note on upgrading)

AI, where to begin?

If you have any interest on AI and related topics you would know the amount of information out there is huge and it drove me crazy reading all the info with nowhere to begin!! So I made this summary for people who are lost and looking for a simple beginning. I will keep updating this post if I find new info:

Continue reading “AI, where to begin?”

AI, where to begin?

Exporting Cassandra 2.2 to Google BigQuery

So we decide to move 5 years of data from Apache Cassandra to Google BigQuery. The problem was not just transferring the data or export/import, the issue was the very old Cassandra!

After extensive research, we have planned the migration to export data to csv and then upload in Google Cloud Storage for importing in Big Query.

The pain was the way Cassandra 1.1 deal with large number of records! There is no pagination so at some point your gonna run out of something! If not mistaken, pagination is introduced since version 2.2.

After all my attempts to upgrade to latest version 3.4 failed I decide to try other versions and luckily the version 2.2 worked! By working I mean I were able to follow the upgrading steps to end and the data were accessible.

Continue reading “Exporting Cassandra 2.2 to Google BigQuery”

Exporting Cassandra 2.2 to Google BigQuery

Pastejacking: what if what you paste is not what you copied!

Those little javascript codes in websites that no one ever check can push notifications and get geolocation with your permission; it can also store files in your cache, open windows, log keystrokes, follow your mouse movements andoverride your clipboard without your permission!

Well, the issue here is that you can not be sure of what you have in your clipboard! I can think of 2 case scenarios that this can be a security issue:

  • When a normal user copy the content from websites and paste directly into a vulnerable software (Microsoft Word?!) and the copied contents simply trigger the vulnerability.
  • When an admin copy and paste some command directly from a website (tutorials?!) into their terminal! This is the creepy one cause depends on the privilege of admin (duh!!) it can download and execute scripts; and then to make things looks OK performs a cleanup and probably do what it supposed to do !!

If you are thinking of disabling javascript, CSS (Cascading Style Sheets) can also be used to hide some contents among what you are copying! The problem with CSS is that you can not be sure what you are copying!

The solution is not that difficult though: just be aware of what you paste, where you paste and think twice before you paste!! Perhaps just paste in a notepad first, or just get a clipboard manager!

P/S: You can find a simple demonstration here:

GitHub – dxa4481/Pastejacking: A demo of overriding what’s in a person’s clipboard

Pastejacking: what if what you paste is not what you copied!

VirusTotal and changes in endpoint security

As you may know, VirusTotal is a Google owned company with huge resources where you can simply get the result of checking a file against multiple AntiVirus (AV) engines.

I recently get to know an announcement from VirusTotal (VT) that probably affects many endpoint security companies. The fact is that VT provides a rich API that enable almost anyone to build their own AV. This easily could be misused by startups endpoint security companies who simply did not have a proper engine but thanks to VT (and hard work of all the powerful AV engines in VT) could simply get a very good detection rate.

During my experience with AV engines it always bothered me that how easy it is to fool everyone without even having a proper engine! Well, now with recent announcement I think this issue is resolved as it strictly forces AV companies to share their engines with VT if they are going to use community’s results.

Perhaps we could expect some of the wrongly praised companies go down since they are no longer able to access VTs results; and most probably have no presentable AV engine to share with the community in return!

VirusTotal and changes in endpoint security

Do we take security seriously?

No! As a matter of fact no one does and that is why even the biggest enterprises get hacked at least once! Security is like a submarine that even smallest holes can let the water to get in the submarine.

I just went through a very interesting attack. It is interesting not because of any complicated attack but the target of the attack! This is actually an old news but interesting to find the details of how Hacking-Team got hacked!

You can find it here: https://pastebin.com/raw/0SNSvyjJ

What have the target done wrong? I would say not taking the security seriously. This is a little reminder that security is a culture and no matter how much we know about it we may actually get hacked (or in this case assist the hack) with our rookie mistakes. You can go through it and see the main defence of the target was their network edge and once the hacker passed it, no more serious security exist!

Do we take security seriously?

Linux restricted administration

One of the challenges when adding a user in Linux environment is when you need to precisely define what they can or can not do. Some may find configuring authorisation in Linux a bit complicated. In my case I needed to add a user with privilege to execute some commands with sudo but without full root access.

The first thing is to create the user we want to customise:
adduser jack

and then create the key and later pass the id_rsa to him:

Continue reading “Linux restricted administration”

Linux restricted administration

WPA/WPA2 Cracking with GPU in AWS

DISCLAIMER: The information provided on this post is to be used for educational purposes only. The website creator is in no way responsible for any misuse of the information provided. All of the information in this website is meant to help the reader develop a hacker defence attitude in order to prevent the attacks discussed. In no way should you use the information to cause any kind of damage directly or indirectly. You implement the information given at your own risk.

In this post we are going to realise how practical it is to perform a brute force attack on a WPA or WPA2 captured handshake. A couple of years ago WPA/WPA2 considered secure but with the current power and cost of cloud computing anyone with slightest interest can setup a super fast server for brute force attempts with very cheap price (as low as $0.6 per hour!).

I am going to walk through my experiment and share the details and results with you. There are dozens of tutorials for this out there but this is just my own little experiment.

Brute forcing a WPA or WPA2 password begins with capturing the 4way handshake of the target WiFi. I am not going to go there as you can find a lot of solutions for that! I can only mention Kali toolbox which provides you the tools. So we will assume you got the WPA 4way handshake in handshake.cap file.

Continue reading “WPA/WPA2 Cracking with GPU in AWS”

WPA/WPA2 Cracking with GPU in AWS

Free secure backup in MEGA

http://mega.nz/ is a secured cloud storage that gives away up to 50GB free space. Using this service is recommended due to very tight security as even the user will not be able to gain access to data if he lose the password (and lose the recovery key).

 

Mega provided some scripts for uploading, syncing and etc to their cloud. This is specially useful when it comes to cheap secure backup of your files. All you need to do is creating a free account for beginning and perhaps purchase a premium account for a better service.

First it is just appropriate to setup proper locale variables:

Continue reading “Free secure backup in MEGA”

Free secure backup in MEGA

Monitoring SMTP server (using mailgraph, qshape and postqueue

Monitoring SMTP mails never been easier! You can check the number of sent emails, bounced emails, rejected emails and etc (you can see demo at http://www.stat.ee.ethz.ch/mailgraph.cgi).

Let’s get the source:

and install dependencies:

Continue reading “Monitoring SMTP server (using mailgraph, qshape and postqueue”

Monitoring SMTP server (using mailgraph, qshape and postqueue