Reported on the forums (http://hpccsystems.com/bb/viewtopic.php?t=513&p=2313#p2313):
I have an intermittent (30% of the time) esp crash when submitting Roxie queries. Here is the crash log:
00000037 2012-09-07 21:56:04 14020 14057 "buffer_key=1"
00000038 2012-09-07 21:56:04 14020 14057 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000039 2012-09-07 21:56:04 14020 14057 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000003A 2012-09-07 21:56:04 14020 14058 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
0000003B 2012-09-07 21:56:04 14020 14059 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
0000003C 2012-09-07 21:56:04 14020 14058 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000003D 2012-09-07 21:56:04 14020 14059 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000003E 2012-09-07 21:56:04 14020 14060 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000041 2012-09-07 21:56:04 14020 14063 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000042 2012-09-07 21:56:04 14020 14063 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000043 2012-09-07 21:56:04 14020 14060 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000003F 2012-09-07 21:56:04 14020 14061 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000044 2012-09-07 21:56:04 14020 14061 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000045 2012-09-07 21:56:04 14020 14064 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000046 2012-09-07 21:56:04 14020 14064 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000040 2012-09-07 21:56:04 14020 14062 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
00000047 2012-09-07 21:56:04 14020 14062 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000048 2012-09-07 21:56:04 14020 14065 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000049 2012-09-07 21:56:04 14020 14065 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000004A 2012-09-07 21:56:04 14020 14066 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
0000004B 2012-09-07 21:56:04 14020 14066 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000004C 2012-09-07 21:56:04 14020 14067 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
0000004D 2012-09-07 21:56:04 14020 14067 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000004E 2012-09-07 21:56:04 14020 14068 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
0000004F 2012-09-07 21:56:04 14020 14068 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000050 2012-09-07 21:56:04 14020 14069 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000051 2012-09-07 21:56:04 14020 14069 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000052 2012-09-07 21:56:04 14020 14070 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
00000053 2012-09-07 21:56:04 14020 14070 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000054 2012-09-07 21:56:04 14020 14071 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
00000055 2012-09-07 21:56:04 14020 14071 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000056 2012-09-07 21:56:04 14020 14072 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
00000057 2012-09-07 21:56:04 14020 14072 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000058 2012-09-07 21:56:04 14020 14073 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
00000059 2012-09-07 21:56:04 14020 14073 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000005A 2012-09-07 21:56:04 14020 14074 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
0000005B 2012-09-07 21:56:04 14020 14074 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000005C 2012-09-07 21:56:04 14020 14075 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country HTTP/1.1"
0000005D 2012-09-07 21:56:04 14020 14075 "POST /WsEcl/json/query/myroxie/search_top_companies_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
0000005E 2012-09-07 21:56:04 14020 14076 "HTTP First Line: POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country HTTP/1.1"
0000005F 2012-09-07 21:56:04 14020 14076 "POST /WsEcl/json/query/myroxie/search_top_cities_by_revenue_given_sic4_and_country, from unknown@172.16.1.245"
00000060 2012-09-07 21:56:04 14020 14057 "soap from json req: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"> <soap:Body><search_top_companies_by_revenue_given_sic4_and_countryRequest><us_sic4>2911</us_sic4><country_id>76</country_id></search_top_companies_by_revenue_given_sic4_and_countryRequest></soap:Body></soap:Envelope>"
00000061 2012-09-07 21:56:05 14020 14070 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000062 2012-09-07 21:56:05 14020 14071 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000063 2012-09-07 21:56:05 14020 14072 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000064 2012-09-07 21:56:05 14020 14073 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000065 2012-09-07 21:56:05 14020 14074 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000066 2012-09-07 21:56:05 14020 14075 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000067 2012-09-07 21:56:05 14020 14059 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000068 2012-09-07 21:56:05 14020 14076 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
00000069 2012-09-07 21:56:05 14020 14058 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
0000006A 2012-09-07 21:56:05 14020 14058 "soap from json req: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"> <soap:Body><search_top_cities_by_revenue_given_sic4_and_countryRequest><us_sic4>2911</us_sic4><country_id>76</country_id></search_top_cities_by_revenue_given_sic4_and_countryRequest></soap:Body></soap:Envelope>"
0000006B 2012-09-07 21:56:05 14020 14057 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
0000006C 2012-09-07 21:56:06 14020 14072 "WARNING: Excessive concurrent Dali SDS client transactions. Transaction delayed."
0000006D 2012-09-07 21:56:06 14020 14059 "soap from json req: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"> <soap:Body><search_top_cities_by_revenue_given_sic4_and_countryRequest><us_sic4>2911</us_sic4><country_id>76</country_id></search_top_cities_by_revenue_given_sic4_and_countryRequest></soap:Body></soap:Envelope>"
0000006E 2012-09-07 21:56:06 14020 14072 "soap from json req: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"> <soap:Body><search_top_cities_by_revenue_given_sic4_and_countryRequest><us_sic4>2911</us_sic4><country_id>76</country_id></search_top_cities_by_revenue_given_sic4_and_countryRequest></soap:Body></soap:Envelope>"
0000006F 2012-09-07 21:56:06 14020 14057 "================================================"
00000070 2012-09-07 21:56:06 14020 14057 "Signal: 11 Segmentation fault"
00000071 2012-09-07 21:56:06 14020 14057 "Fault IP: 0000003D4B079B60"
00000072 2012-09-07 21:56:06 14020 14057 "Accessing: 0000000000000000"
00000073 2012-09-07 21:56:06 14020 14057 "Registers:"
00000074 2012-09-07 21:56:06 14020 14057 "EAX:6569786F00323233 EBX:0000000000000001 ECX:0000000000000005 EDX:0000000000000000 ESI:00002AAAB4007D55 EDI:6569786F00323233"
00000075 2012-09-07 21:56:06 14020 14057 "CS:EIP:0033:0000003D4B079B60"
00000076 2012-09-07 21:56:06 14020 14057 " ESP:0000000054267888 EBP:6569786F00323233"
00000077 2012-09-07 21:56:06 14020 14057 "Stack[0000000054267888]: 00002AAAABAA8824 0000000000002AAA 0000000000000000 FFFF000000000000 83E4D20AFFFF0000 0000000083E4D20A 0000000000000000 B40025E000000000"
00000078 2012-09-07 21:56:06 14020 14057 "Stack[00000000542678A8]: 00002AAAB40025E0 B400829000002AAA 00002AAAB4008290 B4007D2000002AAA 00002AAAB4007D20 0000000000002AAA 0000000000000000 FFFF000000000000"
00000079 2012-09-07 21:56:06 14020 14057 "Stack[00000000542678C8]: 83E4D20AFFFF0000 5426000083E4D20A 0000000054260000 B4007C6800000000 00002AAAB4007C68 B40039F000002AAA 00002AAAB40039F0 0000003900002AAA"
0000007A 2012-09-07 21:56:06 14020 14057 "Stack[00000000542678E8]: 0000004000000039 0000000100000040 0000000000000001 0000000200000000 0000000000000002 B400253000000000 00002AAAB4002530 ABAA43D600002AAA"
0000007B 2012-09-07 21:56:06 14020 14057 "Stack[0000000054267908]: 00002AAAABAA43D6 0000000100002AAA 0000000000000001 0000000100000000 0000000000000001 54267A7000000000 0000000054267A70 4B0325BE00000000"
0000007C 2012-09-07 21:56:06 14020 14057 "Stack[0000000054267928]: 0000003D4B0325BE B40073F00000003D 00002AAAB40073F0 F95A010100002AAA 00002AD2F95A0101 B400449000002AD2 00002AAAB4004490 B4007C6000002AAA"
0000007D 2012-09-07 21:56:06 14020 14057 "Stack[0000000054267948]: 00002AAAB4007C60 B4007C6000002AAA 00002AAAB4007C60 0000000800002AAA 0000000000000008 ABAA43B000000000 00002AAAABAA43B0 0000000200002AAA"
0000007E 2012-09-07 21:56:06 14020 14057 "Stack[0000000054267968]: 0000000000000002 0000000400000000 0000000000000004 54267A7000000000 0000000054267A70 0000000200000000 0000000000000002 B4007C7000000000"
0000007F 2012-09-07 21:56:06 14020 14057 "Backtrace:"
00000080 2012-09-07 21:56:06 14020 14067 "soap from json req: <?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"> <soap:Body><search_top_companies_by_revenue_given_sic4_and_countryRequest><us_sic4>2911</us_sic4><country_id>76</country_id></search_top_companies_by_revenue_given_sic4_and_countryRequest></soap:Body></soap:Envelope>"
00000081 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x2ad2f954de06]"
00000082 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libjlib.so(_Z13excsighandleriP7siginfoPv+0x295) [0x2ad2f954ee55]"
00000083 2012-09-07 21:56:06 14020 14057 " /lib64/libpthread.so.0 [0x2ad2fae72b70]"
00000084 2012-09-07 21:56:06 14020 14057 " /lib64/libc.so.6(strlen+0x10) [0x3d4b079b60]"
00000085 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libdllserver.so(_ZN11DllLocation13queryLocationEv+0xa4) [0x2aaaabaa8824]"
00000086 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libdllserver.so(_Z14orderLocationsPP10IInterfaceS1_+0x26) [0x2aaaabaa43d6]"
00000087 2012-09-07 21:56:06 14020 14057 " /lib64/libc.so.6 [0x3d4b0325be]"
00000088 2012-09-07 21:56:06 14020 14057 " /lib64/libc.so.6 [0x3d4b03246d]"
00000089 2012-09-07 21:56:06 14020 14057 " /lib64/libc.so.6(qsort+0x291) [0x3d4b0329f1]"
0000008A 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libdllserver.so(_ZN8DllEntry15getBestLocationEv+0x7f) [0x2aaaabaa608f]"
0000008B 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libdllserver.so(_ZN9DllServer12getBestMatchEPKc+0x1f) [0x2aaaabaa537f]"
0000008C 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libdllserver.so(_ZN9DllServer14getBestMatchExEPKc+0x16) [0x2aaaabaa5476]"
0000008D 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libdllserver.so(_ZN9DllServer7loadDllEPKc15DllLocationType+0x2c) [0x2aaaabaa5d9c]"
0000008E 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libwuwebview.so(_ZN9WuWebView7loadDllEb+0x85) [0x2aaab1888115]"
0000008F 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libwuwebview.so(_ZN9WuWebView13expandResultsEPKcR12StringBufferj+0x56) [0x2aaab1889a56]"
00000090 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libws_ecl.so(_ZN13CWsEclBinding14handleHttpPostEP12CHttpRequestP13CHttpResponse+0x564) [0x2aaab2625f34]"
00000091 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer14processRequestEv+0x42a) [0x2ad2f9d2ee2a]"
00000092 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libesphttp.so(_ZN11CHttpThread9onRequestEv+0x19b) [0x2ad2f9d29d7b]"
00000093 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libesphttp.so(_ZN18CEspProtocolThread3runEv+0x1a) [0x2ad2f9d590ba]"
00000094 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread5beginEv+0x37) [0x2ad2f95d8277]"
00000095 2012-09-07 21:56:06 14020 14057 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread11_threadmainEPv+0x1f) [0x2ad2f95d8def]"
00000096 2012-09-07 21:56:06 14020 14057 " /lib64/libpthread.so.0 [0x2ad2fae6a73d]"
00000097 2012-09-07 21:56:06 14020 14057 " /lib64/libc.so.6(clone+0x6d) [0x3d4b0d44bd]"
00000098 2012-09-07 21:56:06 14020 14057 "ThreadList:
4225F940 1109784896 14021: CMPNotifyClosedThread
44260940 1143343424 14022: MP Connection Thread
48262940 1210460480 14024: CSocketSelectThread
46261940 1176901952 14025: CMemoryUsageReporter
4A263940 1244019008 14026: unknown
4C264940 1277577536 14027: unknown
4E265940 1311136064 14029: unknown
50266940 1344694592 14030: unknown
52267940 1378253120 14031: CSocketSelectThread
54268940 1411811648 14057: CEspProtocolThread
56269940 1445370176 14058: CEspProtocolThread
5826A940 1478928704 14059: CEspProtocolThread
5A26B940 1512487232 14060: CEspProtocolThread
5C26C940 1546045760 14061: CEspProtocolThread
5E26D940 1579604288 14062: CEspProtocolThread
6026E940 1613162816 14063: CEspProtocolThread
6226F940 1646721344 14064: CEspProtocolThread
64270940 1680279872 14065: CEspProtocolThread
66271940 1713838400 14066: CEspProtocolThread
68272940 1747396928 14067: CEspProtocolThread
6A273940 1780955456 14068: CEspProtocolThread
6C274940 1814513984 14069: CEspProtocolThread
6E275940 1848072512 14070: CEspProtocolThread
70276940 1881631040 14071: CEspProtocolThread
72277940 1915189568 14072: CEspProtocolThread
74278940 1948748096 14073: CEspProtocolThread
76279940 1982306624 14074: CEspProtocolThread
7827A940 2015865152 14075: CEspProtocolThread
7A27B940 2049423680 14076: CEspProtocolThread
"
This was taken during a stress test, where another system was executing a pair of json Roxie queries (via curl) 20 times in the background, which is to say simultaneously as bash would allow. Sometimes all queries succeed, other times a segfault occurs, and when esp segfaults it could do it on the first processed query or any of them.
Also seeing
0000012D 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_Z16PrintStackReportv+0x26) [0x2af2c7fbee06]"
0000012E 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_Z13excsighandleriP7siginfoPv+0x295) [0x2af2c7fbfe55]"
0000012F 2012-09-07 22:26:13 24942 25099 " /lib64/libpthread.so.0 [0x2af2c98e3b70]"
00000130 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_ZN9InitTable4exitEPv+0x6f) [0x2af2c7fdeacf]"
00000131 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_Z16FreeSharedObjectPv+0x9) [0x2af2c8057119]"
00000132 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_ZN12SharedObject6unloadEv+0x25) [0x2af2c8057155]"
00000133 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libdllserver.so(_ZN9HelperDllD0Ev+0x5a) [0x2aaaabaaaa7a]"
00000134 2012-09-07 22:26:13 24942 25099 " esp(_ZNK10CInterface7ReleaseEv+0x3c) [0x41050c]"
00000135 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libdllserver.so(_ZNK9HelperDll7ReleaseEv+0x9) [0x2aaaabaaafd9]"
00000136 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libwuwebview.so(_ZN9WuWebViewD0Ev+0xb4) [0x2aaab188c874]"
00000137 2012-09-07 22:26:13 24942 25099 " esp(_ZNK10CInterface7ReleaseEv+0x3c) [0x41050c]"
00000138 2012-09-07 22:26:13 24942 25111 "TxSummary[activeReqs=17;user=@172.16.1.245;total=2159ms;]"
00000139 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libwuwebview.so(_ZNK9WuWebView7ReleaseEv+0x9) [0x2aaab188aef9]"
0000013A 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libws_ecl.so(_ZN13CWsEclBinding14handleHttpPostEP12CHttpRequestP13CHttpResponse+0x577) [0x2aaab2626f47]"
0000013B 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libesphttp.so(_ZN14CEspHttpServer14processRequestEv+0x42a) [0x2af2c879fe2a]"
0000013C 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libesphttp.so(_ZN11CHttpThread9onRequestEv+0x19b) [0x2af2c879ad7b]"
0000013D 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libesphttp.so(_ZN18CEspProtocolThread3runEv+0x1a) [0x2af2c87ca0ba]"
0000013E 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread5beginEv+0x37) [0x2af2c8049277]"
0000013F 2012-09-07 22:26:13 24942 25099 " /opt/HPCCSystems/lib/libjlib.so(_ZN6Thread11_threadmainEPv+0x1f) [0x2af2c8049def]"
...
It appears that modifying the configuration so that ESP spawns at most only one concurrent thread (unlimited concurrent threads is the default) is an effective workaround. At least, no segfaults appear.
@dcamper
Reported on the forums (http://hpccsystems.com/bb/viewtopic.php?t=513&p=2313#p2313):
I have an intermittent (30% of the time) esp crash when submitting Roxie queries. Here is the crash log:
This was taken during a stress test, where another system was executing a pair of json Roxie queries (via curl) 20 times in the background, which is to say simultaneously as bash would allow. Sometimes all queries succeed, other times a segfault occurs, and when esp segfaults it could do it on the first processed query or any of them.
Also seeing
...
It appears that modifying the configuration so that ESP spawns at most only one concurrent thread (unlimited concurrent threads is the default) is an effective workaround. At least, no segfaults appear.
@dcamper