Contact Us
SaaS Log InXRay Login
blog

Running Solr 6.6 in Production

Nastel Technologies®
May 28, 2020

I thought I’d share our experiences of running Solr in production. So far Solr 6.6.6 has been very stable provided GC, memory and other system resources are carefully managed. Here are some of the technical details which I hope will be useful to anyone considering this platform.

We have been running Solr 6.6.6 with Java 9 in production for about a year now with the data index size of about 1.5 TB per node.

This cluster is a 4 node cluster indexing about 30-50GB of data per day with time to live (TTL) ranging from 14 to 90 days.

Here is how we configured Solr 6 with Java 9

1) Default Solr GC settings caused OOM exceptions as index size grew. We had to switch to G1 collector to improve stability. Solr would periodically stop with OOM with default GC settings.

2) Index size will impact max heap size requirement for Solr. Of course, a lot depends on your data and your data & SOLR configuration. We disabled most caches and even then heap size would grow with growth in index size. We found that ratio of index to heap to be around 23-25x.

Meaning estimated heap size per Solr node: Size of Index per node divided by 23. For 1TB index, you would expect to have heap at around 44GB. Experiment with your configuration for optimum settings.

3) G1 collector settings had to be tweaked carefully. Here are our settings for data index size of around 1.5TB:

SOLR_JAVA_MEM=”-Xmx64g -Xms64g”
GC_TUNE=” \
-XX:+UseG1GC \
-XX:+AggressiveOpts \
-XX:+PerfDisableSharedMem \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=32m \
-XX:MaxGCPauseMillis=500 \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/opt/nastel/solr/var/logs

Below are Max Heap and G1HeapRegionSize combinations:

4GB — 2MB
8GB — 4MB
16GB — 8MB
32GB — 16MB
64GB — 32MB

4) Separate filesystems for Solr binaries and /opt/nastel/solr/var where all logs and data are stored. We mounted /opt/nastel/solr/var file system on high performance direct attached SSD drives.

Solr VAR filesystem is sized as follows:
Total expected index size * 2. Solr will need significantly more space during restart recovery. You will run out of disk space if you don’t have enough free space available during restart recovery.

5) VM RAM size is Solr max heap size * 2 (at least 2x) to make sure Solr can store as much index into memory as possible. We typically run on 128GB RAM Linux VM if Solr heap size is 64GB.

 

Nastel Technologies is the global leader in Integration Infrastructure Management (i2M). It helps companies achieve flawless delivery of digital services powered by integration infrastructure by delivering tools for Middleware Management, Monitoring, Tracking, and Analytics to detect anomalies, accelerate decisions, and enable customers to constantly innovate, to answer business-centric questions, and provide actionable guidance for decision-makers. It is particularly focused on IBM MQ, Apache Kafka, Solace, TIBCO EMS, ACE/IIB and also supports RabbitMQ, ActiveMQ, Blockchain, IOT, DataPower, MFT, IBM Cloud Pak for Integration and many more.

 

The Nastel i2M Platform provides:

Author

  • Albert Mavashev

    Innovator, Growth Hacker, Idea Generator. As Nastel’s CTO, Albert turns ideas into real-world solutions. Albert’s specialties include Integration Infrastructure Management, Application Performance Management, IT Service Management, Transaction Analytics, Performance Measurement & Practices, Streaming Analytics @ Scale, Clustered Computing, Big & Fast Data, Complex Event Processing (CEP), Messaging & Middleware, Ops & DevOps, Blockchain, cryptocurrencies, digital tokens/assets, and building & scaling & analyzing digital de-centralized economies.

Comments

Write a comment
Leave a Reply
Your email address will not be published. Required fields are marked *
Comment * This field is required!
First name * This field is required!
Email * Please, enter valid email address!

Schedule your Meeting

 


Schedule your Meeting


Subscribe

Schedule a Meeting to Learn More