INFO: Using directory: "1" INFO: Using existing c-file: ./Test_TasReso.c INFO: Using existing binary: ./Test_TasReso.out INFO: === Simulation 'Test_TasReso' (/home/nexmap/pkwi/TESTS/2023-10-01/McStas_8GPU_5e7/Test_TasReso/Test_TasReso.instr): running on 7 nodes (master is 'hypatia.fysik.dtu.dk', MPI version 3.1). Have found 8 GPU devices on rank 4. Will use device 4. Have found 8 GPU devices on rank 1. Will use device 1. Have found 8 GPU devices on rank 6. Will use device 6. Have found 8 GPU devices on rank 0. Will use device 0. -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- [Test_TasReso] Initialize -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- [Test_TasReso] Initialize Opening input file '/u/data/pkwi/McStas/mcstas/3.x-dev/data/HOPG.rfl' (Table_Read_Offset) Opening input file '/u/data/pkwi/McStas/mcstas/3.x-dev/data/HOPG.trm' (Table_Read_Offset) Incoherent: sample_Vana: Vc=13.827 [Angs] sigma_abs=5.08 [barn] sigma_inc=5.08 [barn] Mode: lengths and angles Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] Mode: lengths and angles Reusing input file 'HOPG.rfl' (Table_Read_Offset) Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reusing input file 'HOPG.trm' (Table_Read_Offset) Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] Have found 8 GPU devices on rank 5. Will use device 5. -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- [Test_TasReso] Initialize Mode: lengths and angles Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] Have found 8 GPU devices on rank 2. Will use device 2. -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- [Test_TasReso] Initialize [Test_TasReso] Initialize [Test_TasReso] Initialize Mode: lengths and angles Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] Mode: lengths and angles Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] Mode: lengths and angles Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] Have found 8 GPU devices on rank 3. Will use device 3. -------------------------------------------------------------------------------- Source: lambda = 4.5000 A, k = 1.3963 1/A, E = 4.0397 meV Monochromator: ki = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Analyser: kf = 1.3963, theta = 42.1165 deg, d = 3.3550, vfoc = 0.6706 m, hfoc = 1.4911 m Sample: num = 2, theta = 26.7400, 2theta = 53.4800 -------------------------------------------------------------------------------- [Test_TasReso] Initialize Mode: lengths and angles Single_crystal: TasResoTest_xtal.laz structure a=5 b=5 c=5 aa=90 bb=90 cc=90 V0=125 Single_crystal: sample_xtal: Read 6 reflections from file 'TasResoTest_xtal.laz' Single_crystal: sample_xtal: Vc=125 [Angs] sigma_abs=0 [barn] sigma_inc=0 [barn] reflections=TasResoTest_xtal.laz Direct space lattice orientation: a = [0 5 3.06162e-16] b = [0 0 5] c = [5 3.06162e-16 3.06162e-16] Reciprocal space lattice orientation: a* = [-7.69468e-17 1.25664 0] b* = [-7.69468e-17 -7.69468e-17 1.25664] c* = [1.25664 0 0] *** TRACE end *** Save [Test_TasReso] Detector: psdmon_I=994.677 psdmon_ERR=57.4757 psdmon_N=573 "psd.dat" Detector: divmon_I=961.454 divmon_ERR=56.3555 divmon_N=554 "div.dat" Detector: lmon_I=994.677 lmon_ERR=57.4757 lmon_N=573 "lam.dat" Detector: emon_I=994.677 emon_ERR=57.4757 emon_N=573 "E.dat" : 4.04668 meV , E-width : 0.057027 meV : 4.03489 meV , E-width : 0.0696384 meV : 4.02748 meV , E-width : 0.0545461 meV : 4.03548 meV , E-width : 0.0575418 meV : 4.04827 meV , E-width : 0.0506855 meV : 4.0415 meV , E-width : 0.0498454 meV Events: "reso.dat" : 4.04672 meV , E-width : 0.0667102 meV [hypatia:20024:0:20024] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) [hypatia:20026:0:20026] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) [hypatia:20028:0:20028] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) [hypatia:20030:0:20030] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) [hypatia:20032:0:20032] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) [hypatia:20033:0:20033] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) ==== backtrace (tid: 20024) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7f8efa6a3364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7f8efa6a368c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7f8efa6a38fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7f8efc33d630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] ==== backtrace (tid: 20026) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7fc3ebb9c364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7fc3ebb9c68c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7fc3ebb9c8fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7fc3ed836630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] ==== backtrace (tid: 20028) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7f4650a55364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7f4650a5568c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7f4650a558fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7f46526ef630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4651cb7555] 10 ./Test_TasReso.out() [0x4032a9] ================================= ==== backtrace (tid: 20030) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7fa4488af364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7fa4488af68c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7fa4488af8fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7fa44a549630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fa449b11555] 10 ./Test_TasReso.out() [0x4032a9] ================================= ==== backtrace (tid: 20032) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7f4e12aee364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7f4e12aee68c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7f4e12aee8fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7f4e14788630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f4e13d50555] 10 ./Test_TasReso.out() [0x4032a9] ================================= ==== backtrace (tid: 20033) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7fcab4f41364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7fcab4f4168c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7fcab4f418fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7fcab6bdb630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fcab61a3555] 10 ./Test_TasReso.out() [0x4032a9] ================================= 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8efb905555] 10 ./Test_TasReso.out() [0x4032a9] ================================= 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc3ecdfe555] 10 ./Test_TasReso.out() [0x4032a9] ================================= [hypatia:20034:0:20034] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2073ff9c8) ==== backtrace (tid: 20034) ==== 0 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(ucs_handle_error+0x124) [0x7f7639062364] 1 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x3268c) [0x7f763906268c] 2 /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/lib/libucs.so.0(+0x328fb) [0x7f76390628fb] 3 /lib64/libpthread.so.0(+0xf630) [0x7f763acfc630] 4 ./Test_TasReso.out() [0x434dbb] 5 ./Test_TasReso.out() [0x43642e] 6 ./Test_TasReso.out() [0x467a17] 7 ./Test_TasReso.out() [0x4691c0] 8 ./Test_TasReso.out() [0x4181b9] 9 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f763a2c4555] 10 ./Test_TasReso.out() [0x4032a9] ================================= /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20034 Segmentation fault "$@" -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20026 Segmentation fault (core dumped) "$@" /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20032 Segmentation fault (core dumped) "$@" /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20033 Segmentation fault "$@" /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20024 Segmentation fault (core dumped) "$@" /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20028 Segmentation fault (core dumped) "$@" /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind: line 12: 20030 Segmentation fault "$@" -------------------------------------------------------------------------- mpirun detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[47874,1],6] Exit code: 139 -------------------------------------------------------------------------- INFO: call to /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/bin/mpirun failed with Command '/opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/bin/mpirun -np 7 /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind ./Test_TasReso.out --seed 1000 --ncount 50000000.0 --dir 1 --format McCode src_lam=4.5' returned non-zero exit status 139. Traceback (most recent call last): File "/u/data/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mcrun.py", line 562, in main() File "/u/data/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mcrun.py", line 540, in main mcstas.run() # in mccode.py File "/u/data/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 251, in run return self.runMPI(args, pipe, override_mpi) File "/u/data/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 280, in runMPI return Process(binpath).run(args, pipe=pipe) File "/u/data/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 71, in run raise err File "/u/data/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 67, in run proc = run(command, shell=True, check=True, text=True, capture_output=pipe) File "/u/data/pkwi/McStas/mcstas/3.x-dev/miniconda3/lib/python3.10/subprocess.py", line 524, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '/opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/12.2/openmpi4/openmpi-4.1.5/bin/mpirun -np 7 /u/data/pkwi/McStas/mcstas/3.x-dev/bin/acc_gpu_bind ./Test_TasReso.out --seed 1000 --ncount 50000000.0 --dir 1 --format McCode src_lam=4.5' returned non-zero exit status 139.