<div class="container">
<h1>Feature update for slurm_23_02 and pdsh</h1>
<table class="table table-striped table-bordered">
<tbody>
<tr>
<th>Announcement ID:</th>
<td>SUSE-FU-2023:3322-1</td>
</tr>
<tr>
<th>Rating:</th>
<td>moderate</td>
</tr>
<tr>
<th>References:</th>
<td>
<ul>
<li style="display: inline;">
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1088693">#1088693</a>
</li>
<li style="display: inline;">
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1206795">#1206795</a>
</li>
<li style="display: inline;">
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1208846">#1208846</a>
</li>
<li style="display: inline;">
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1209216">#1209216</a>
</li>
<li style="display: inline;">
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1209260">#1209260</a>
</li>
<li style="display: inline;">
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1212946">#1212946</a>
</li>
<li style="display: inline;">
<a href="https://jira.suse.com/browse/PED-2987">PED-2987</a>
</li>
</ul>
</td>
</tr>
<tr>
<th>Affected Products:</th>
<td>
<ul class="list-group">
<li class="list-group-item">openSUSE Leap 15.4</li>
<li class="list-group-item">openSUSE Leap 15.5</li>
<li class="list-group-item">SUSE Linux Enterprise High Performance Computing 15 SP1</li>
<li class="list-group-item">SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1</li>
</ul>
</td>
</tr>
</tbody>
</table>
<p>An update that contains one feature and has six feature fixes can now be installed.</p>
<h2>Description:</h2>
<p>This update for slurm_23_02 and pdsh fixes the following issues:</p>
<p>slurm_23_02 - New version upgrade of Slurm to 23.02 (jsc#PED-2987):</p>
<ul>
<li>For the full list of new features and changes please consult the packaged NEWS file and the following references:</li>
<li>23.02.2</li>
<li>23.02.1</li>
<li>23.02.0</li>
<li>Important notes:</li>
<li>If using the <code>slurmdbd</code> (Slurm DataBase Daemon) you must update this first.</li>
<li>If using a backup DBD you must start the primary first to do any database conversion, the backup will not start
until this has happened.</li>
<li>The 23.02 <code>slurmdbd</code> will work with Slurm daemons of version 21.08 and above. You will not need to update all
clusters at the same time, but it is very important to update <code>slurmdbd</code> first and having it running before updating
any other clusters making use of it.</li>
<li>Slurm can be upgraded from version 21.08 or 22.05 to version 23.02 without loss of jobs or other state information.
Upgrading directly from an earlier version of Slurm will result in loss of state information.</li>
<li>All SPANK plugins must be recompiled when upgrading from any Slurm version prior to 23.02</li>
<li>PMIx v1.x is no longer supported</li>
<li>Packaging patches and changes:</li>
<li>Only call slurm_init() if Slurm > 21.02 (bsc#1212946)</li>
<li>Web-configurator: changed presets to SUSE defaults.</li>
<li>Use libpmix.so.2 instead of libpmix.so to fix (bsc#1209260) this removes the need of pmix-pluginlib</li>
<li><code>slurm-plugins</code> need to require <code>pmix-pluginlib</code> (bsc#1209260)</li>
<li>Remove workaround to fix the restart issue in an Slurm package described in bsc#1088693
The Slurm version in this package is 16.05. Any attempt to directly migrate to the current version is bound to fail</li>
<li>Now require <code>slurm-munge</code> if <code>munge</code> authentication is installed</li>
<li>testsuite: on later SUSE versions claim ownership of directory <code>/etc/security/limits.d</code></li>
<li>Move the ext_sensors/rrd plugin to a separate package:
this plugin requires <code>librrd</code> which in turn requires huge parts of the client side X Window System stack.
There is probably no use in cluttering up a system for a plugin that probably only used by a few</li>
<li>Configuration file changes:</li>
<li><code>job_container.conf</code> - Added "<code>Dirs</code>" option to list desired private mount points</li>
<li><code>node_features</code> plugins - invalid users specified for <code>AllowUserBoot</code> will now result in <code>fatal()</code> rather than just
an error</li>
<li>Allow jobs to queue even if the user is not in <code>AllowGroups</code> when <code>EnforcePartLimits=no</code> is set. This ensures
consistency for all the Partition access controls, and matches the documented behavior for <code>EnforcePartLimits</code></li>
<li>Add <code>InfluxDBTimeout</code> parameter to <code>acct_gather.conf</code></li>
<li><code>job_container/tmpfs</code> - add support for expanding <code>%h</code> and <code>%n</code> in <code>BasePath</code></li>
<li><code>slurm.conf</code> - Removed <code>SlurmctldPlugstack</code> option</li>
<li>Add new <code>SlurmctldParameters=validate_nodeaddr_threads=<number></code> option to allow concurrent hostname resolution at
<code>slurmctld</code> startup</li>
<li>Add new <code>AccountingStoreFlags=job_extra</code> option to store a job's extra field in the database</li>
<li>Add new "<code>defer_batch</code>" option to <code>SchedulerParameters</code> to only defer scheduling for batch jobs</li>
<li>Add new <code>DebugFlags</code> option '<code>JobComp</code>' to replace '<code>Elasticsearch</code>'</li>
<li>Add configurable job requeue limit parameter - <code>MaxBatchRequeue</code> - in <code>slurm.conf</code> to permit changes from the old
hard-coded value of 5</li>
<li><code>helpers.conf</code> - Allow specification of node specific features</li>
<li><code>helpers.conf</code> - Allow many features to one helper script</li>
<li><code>job_container/tmpfs</code> - Add "<code>Shared</code>" option to support shared namespaces. This allows autofs to work with the
<code>job_container/tmpfs</code> plugin when enabled</li>
<li><code>acct_gather.conf</code> - Added <code>EnergyIPMIPowerSensors=Node=DCMI</code> and <code>Node=DCMI_ENHANCED</code>.</li>
<li>Add new "<code>getnameinfo_cache_timeout=<number></code>" option to CommunicationParameters to adjust or disable caching the
results of <code>getnameinfo()</code></li>
<li>Add new PrologFlags=ForceRequeueOnFail option to automatically requeue batch jobs on Prolog failures regardless of
the job --requeue setting</li>
<li>Add <code>HealthCheckNodeState=NONDRAINED_IDLE</code> option.</li>
<li>Add '<code>explicit</code>' to Flags in <code>gres.conf</code>. This makes it so the gres is not automatically added to a job's
allocation when <code>--exclusive</code> is used. Note that this is a per-node flag.</li>
<li>Moved the "<code>preempt_</code>" options from <code>SchedulerParameters</code> to <code>PreemptParameters</code>, and dropped the prefix from the
option names. (The old options will still be parsed for backwards compatibility, but are now undocumented.)</li>
<li>Add <code>LaunchParameters=ulimit_pam_adopt</code>, which enables setting <code>RLIMIT_RSS</code> in adopted processes.</li>
<li>Update SwitchParameters=job_vni to enable/disable creating job VNIs for all jobs, or when a user requests them</li>
<li>Update <code>SwitchParameters=single_node_vni</code> to enable/disable creating single node VNIs for all jobs, or when a user
requests them</li>
<li>Add ability to preserve <code>SuspendExc*</code> parameters on reconfig with <code>ReconfigFlags=KeepPowerSaveSettings</code></li>
<li><code>slurmdbd.conf</code> - Add new <code>AllResourcesAbsolute</code> to force all new resources to be created with the <code>Absolute</code> flag</li>
<li><code>topology/tree</code> - Add new <code>TopologyParam=SwitchAsNodeRank</code> option to reorder nodes based on switch layout. This can
be useful if the naming convention for the nodes does not natually map to the network topology</li>
<li>Removed the default setting for <code>GpuFreqDef</code>. If unset, no attempt to change the GPU frequency will be made if
<code>--gpu-freq</code> is not set for the step</li>
<li>Command Changes:</li>
<li><code>sacctmgr</code> - no longer force updates to the AdminComment, Comment, or SystemComment to lower-case</li>
<li><code>sinfo</code> - Add -F/--future option to sinfo to display future nodes.</li>
<li><code>sacct</code> - Rename 'Reserved' field to 'Planned' to match sreport and the nomenclature of the 'Planned' node</li>
<li><code>scontrol</code> - advanced reservation flag MAINT will no longer replace nodes, similar to STATIC_ALLOC</li>
<li><code>sbatch</code> - add parsing for #PBS -d and #PBS -w.</li>
<li><code>scontrol</code> show assoc_mgr will show username(uid) instead of uid in QoS section.</li>
<li>Add <code>strigger --draining</code> and <code>-R/--resume</code> options.</li>
<li>Change <code>--oversubscribe</code> and <code>--exclusive</code> to be mutually exclusive for job submission. Job submission commands
will now fatal if both are set. Previously, these options would override each other, with the last one in the job
submission command taking effect.</li>
<li><code>scontrol</code> - Requested TRES and allocated TRES will now always be printed when showing jobs, instead of one TRES
output that was either the requested or allocated.</li>
<li><code>srun --ntasks-per-core</code> now applies to job and step allocations. Now, use of <code>--ntasks-per-core=1</code> implies
<code>--cpu-bind=cores</code> and <code>--ntasks-per-core>1</code> implies <code>--cpu-bind=threads</code>.</li>
<li><code>salloc/sbatch/srun</code> - Check and abort if <code>ntasks-per-core</code> > <code>threads-per-core</code>.</li>
<li><code>scontrol</code> - Add <code>ResumeAfter=<secs></code> option to "scontrol update nodename=".</li>
<li>Add a new "nodes=" argument to scontrol setdebug to allow the debug level on the slurmd processes to be temporarily
altered</li>
<li>Add a new "nodes=" argument to "scontrol setdebugflags" as well.</li>
<li>Make it so <code>scrontab</code> prints client-side the job_submit() err_msg (which can be set i.e. by using the log_user()
function for the lua plugin).</li>
<li><code>scontrol</code> - Reservations will not be allowed to have STATIC_ALLOC or MAINT flags and REPLACE[_DOWN] flags
simultaneously</li>
<li><code>scontrol</code> - Reservations will only accept one reoccurring flag when being created or updated.</li>
<li><code>scontrol</code> - A reservation cannot be updated to be reoccurring if it is already a floating reservation.</li>
<li><code>squeue</code> - removed unused '%s' and 'SelectJobInfo' formats.</li>
<li><code>squeue</code> - align print format for exit and derived codes with that of other components
(<exit_status>:<signal_number>).</li>
<li><code>sacct</code> - Add --array option to expand job arrays and display array tasks on separate lines.</li>
<li>Partial support for <code>--json</code> and <code>--yaml</code> formated outputs have been implemented for <code>sacctmgr</code>, <code>sdiag</code>, <code>sinfo</code>,
<code>squeue</code>, and <code>scontrol</code>. The resultant data ouput will be filtered by normal command arguments.
Formatting arguments will continue to be ignored.</li>
<li><code>salloc/sbatch/srun</code> - extended the <code>--nodes</code> syntax to allow for a list of valid node counts to be allocated to
the job. This also supports a "step count" value (e.g., --nodes=20-100:20 is equivalent to --nodes=20,40,60,80,100)
which can simplify the syntax when the job needs to scale by a certain "chunk" size</li>
<li><code>srun</code> - add user requestible vnis with '--network=job_vni' option</li>
<li><code>srun</code> - add user requestible single node vnis with the <code>--network=single_node_vni</code> option</li>
<li>API Changes:</li>
<li><code>job_container</code> plugins - <code>container_p_stepd_create()</code> function signature replaced <code>uint32_t</code> uid with
<code>stepd_step_rec_t*</code> step.</li>
<li><code>gres</code> plugins - <code>gres_g_get_devices()</code> function signature replaced <code>pid_t pid</code> with <code>stepd_step_rec_t*</code> step.</li>
<li><code>cgroup</code> plugins - <code>task_cgroup_devices_constrain()</code> function signature removed <code>pid_t pid</code>.</li>
<li><code>task</code> plugins - <code>replace task_p_pre_set_affinity()</code>, <code>task_p_set_affinity()</code>, and <code>task_p_post_set_affinity()</code>
with <code>task_p_pre_launch_priv()</code> like it was back in slurm 20.11.</li>
<li>Allow for concurrent processing of <code>job_submit_g_submit()</code> and <code>job_submit_g_modify()</code> calls. If your plugin is not
capable of concurrent operation you must add additional locking within your plugin.</li>
<li>Removed return value from slurm_list_append().</li>
<li>The List and ListIterator types have been removed in favor of list_t and list_itr_t respectively.</li>
<li>burst buffer plugins:<ul>
<li>add <code>bb_g_build_het_job_script()</code></li>
<li><code>bb_g_get_status()</code> - added authenticated UID and GID</li>
<li><code>bb_g_run_script()</code> - added job_info argument</li>
</ul>
</li>
<li><code>burst_buffer.lua</code> - Pass UID and GID to most hooks. Pass <code>job_info</code> (detailed job information) to many hooks.
See <code>etc/burst_buffer.lua.example</code> for a complete list of changes.
<em>WARNING</em>: Backwards compatibility is broken for <code>slurm_bb_get_status</code>: UID and GID are passed before the variadic
arguments. If UID and GID are not explicitly listed as arguments to <code>slurm_bb_get_status()</code>, then they will be
included in the variadic arguments. Backwards compatibility is maintained for all other hooks because the new
arguments are passed after the existing arguments.</li>
<li><code>node_features plugins</code> changes:<ul>
<li><code>node_features_p_reboot_weight()</code> function removed.</li>
<li><code>node_features_p_job_valid()</code> - added parameter feature_list.</li>
<li><code>node_features_p_job_xlate()</code> - added parameters feature_list and <code>job_node_bitmap</code></li>
</ul>
</li>
<li>New <code>data_parser</code> interface with v0.0.39 plugin</li>
<li>Test Suite fixes:</li>
<li>Update README_Testsuite.md</li>
<li>Clean up left over files when de-installing test suite</li>
<li>Adjustment to test suite package: for SLE mark the openmpi4 devel package and slurm-hdf5 optional</li>
<li>Add <code>-ffat-lto-objects</code> to the build flags when LTO is set to make sure the object files we ship with the test
suite still work correctly.</li>
<li>Improve <code>setup-testsuite.sh</code>: copy ssh fingerprints from all nodes</li>
</ul>
<p>pdsh:</p>
<ul>
<li>Prepared <code>pdsh</code> for Slurm 23.02 (jsc#PED-2987)</li>
<li>Fix slurm plugin: make sure slurm_init() is called before using the Slurm API (bsc#1209216)</li>
<li>Fix regression in Slurm 23.02 breaking the pdsh-internal List type by exposing it thru it's public API (bsc#1208846)</li>
<li>Backport a number of features and fixes (bsc#1206795):</li>
<li>Add '-C' option on Slrum plugin to restrict selected nodes to ones with the specified features present</li>
<li>Add option '-k' to the ssh plugin to fail faster on connection failures</li>
<li>Fix use of <code>strchr</code></li>
<li><code>dshbak</code>: Fix uninitialized use of $tag on empty input</li>
<li><code>dsh</code>: Release a lock that is no longer used in dsh()</li>
</ul>
<h2>Patch Instructions:</h2>
<p>
To install this SUSE Moderate update use the SUSE recommended
installation methods like YaST online_update or "zypper patch".<br/>
Alternatively you can run the command listed for your product:
</p>
<ul class="list-group">
<li class="list-group-item">
openSUSE Leap 15.4
<br/>
<code>zypper in -t patch openSUSE-SLE-15.4-2023-3322=1</code>
</li>
<li class="list-group-item">
openSUSE Leap 15.5
<br/>
<code>zypper in -t patch openSUSE-SLE-15.5-2023-3322=1</code>
</li>
<li class="list-group-item">
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1
<br/>
<code>zypper in -t patch SUSE-SLE-Product-HPC-15-SP1-LTSS-2023-3322=1</code>
</li>
</ul>
<h2>Package List:</h2>
<ul>
<li>
openSUSE Leap 15.4 (aarch64 ppc64le s390x x86_64)
<ul>
<li>pdsh_slurm_20_02-debugsource-2.34-150100.10.19.1</li>
<li>pdsh-slurm_20_02-2.34-150100.10.19.1</li>
<li>pdsh-slurm_20_02-debuginfo-2.34-150100.10.19.1</li>
</ul>
</li>
<li>
openSUSE Leap 15.5 (aarch64 ppc64le s390x x86_64)
<ul>
<li>pdsh_slurm_20_02-debugsource-2.34-150100.10.19.1</li>
<li>pdsh-slurm_20_02-2.34-150100.10.19.1</li>
<li>pdsh-slurm_20_02-debuginfo-2.34-150100.10.19.1</li>
</ul>
</li>
<li>
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (aarch64 x86_64)
<ul>
<li>libnss_slurm2_23_02-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-plugins-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-cray-debuginfo-23.02.2-150100.3.3.1</li>
<li>pdsh_slurm_20_02-debugsource-2.34-150100.10.19.1</li>
<li>slurm_23_02-pam_slurm-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-torque-debuginfo-23.02.2-150100.3.3.1</li>
<li>pdsh-slurm-debuginfo-2.34-150100.10.19.1</li>
<li>libpmi0_23_02-debuginfo-23.02.2-150100.3.3.1</li>
<li>libslurm39-debuginfo-23.02.2-150100.3.3.1</li>
<li>pdsh-slurm_22_05-2.34-150100.10.19.1</li>
<li>slurm_23_02-node-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-sview-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-plugin-ext-sensors-rrd-23.02.2-150100.3.3.1</li>
<li>pdsh-machines-debuginfo-2.34-150100.10.19.1</li>
<li>pdsh-slurm-2.34-150100.10.19.1</li>
<li>pdsh-slurm_23_02-2.34-150100.10.19.1</li>
<li>slurm_23_02-sview-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-munge-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-sql-23.02.2-150100.3.3.1</li>
<li>pdsh-machines-2.34-150100.10.19.1</li>
<li>pdsh_slurm_22_05-debugsource-2.34-150100.10.19.1</li>
<li>perl-slurm_23_02-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-auth-none-debuginfo-23.02.2-150100.3.3.1</li>
<li>pdsh-slurm_23_02-debuginfo-2.34-150100.10.19.1</li>
<li>pdsh-debuginfo-2.34-150100.10.19.1</li>
<li>pdsh-slurm_22_05-debuginfo-2.34-150100.10.19.1</li>
<li>pdsh-dshgroup-debuginfo-2.34-150100.10.19.1</li>
<li>pdsh-genders-2.34-150100.10.19.1</li>
<li>slurm_23_02-debugsource-23.02.2-150100.3.3.1</li>
<li>libslurm39-23.02.2-150100.3.3.1</li>
<li>pdsh-slurm_20_02-2.34-150100.10.19.1</li>
<li>slurm_23_02-pam_slurm-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-rest-23.02.2-150100.3.3.1</li>
<li>pdsh-netgroup-2.34-150100.10.19.1</li>
<li>perl-slurm_23_02-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-rest-debuginfo-23.02.2-150100.3.3.1</li>
<li>libnss_slurm2_23_02-23.02.2-150100.3.3.1</li>
<li>pdsh-2.34-150100.10.19.1</li>
<li>slurm_23_02-slurmdbd-23.02.2-150100.3.3.1</li>
<li>pdsh-dshgroup-2.34-150100.10.19.1</li>
<li>slurm_23_02-auth-none-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-lua-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-munge-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-devel-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-lua-23.02.2-150100.3.3.1</li>
<li>pdsh-debugsource-2.34-150100.10.19.1</li>
<li>slurm_23_02-23.02.2-150100.3.3.1</li>
<li>pdsh-netgroup-debuginfo-2.34-150100.10.19.1</li>
<li>pdsh-genders-debuginfo-2.34-150100.10.19.1</li>
<li>slurm_23_02-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-torque-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-node-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-slurmdbd-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-plugins-debuginfo-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-sql-debuginfo-23.02.2-150100.3.3.1</li>
<li>pdsh-slurm_20_02-debuginfo-2.34-150100.10.19.1</li>
<li>slurm_23_02-cray-23.02.2-150100.3.3.1</li>
<li>libpmi0_23_02-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-plugin-ext-sensors-rrd-debuginfo-23.02.2-150100.3.3.1</li>
</ul>
</li>
<li>
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (noarch)
<ul>
<li>slurm_23_02-webdoc-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-config-man-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-config-23.02.2-150100.3.3.1</li>
<li>slurm_23_02-doc-23.02.2-150100.3.3.1</li>
</ul>
</li>
</ul>
<h2>References:</h2>
<ul>
<li>
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1088693">https://bugzilla.suse.com/show_bug.cgi?id=1088693</a>
</li>
<li>
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1206795">https://bugzilla.suse.com/show_bug.cgi?id=1206795</a>
</li>
<li>
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1208846">https://bugzilla.suse.com/show_bug.cgi?id=1208846</a>
</li>
<li>
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1209216">https://bugzilla.suse.com/show_bug.cgi?id=1209216</a>
</li>
<li>
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1209260">https://bugzilla.suse.com/show_bug.cgi?id=1209260</a>
</li>
<li>
<a href="https://bugzilla.suse.com/show_bug.cgi?id=1212946">https://bugzilla.suse.com/show_bug.cgi?id=1212946</a>
</li>
<li>
<a href="https://jira.suse.com/browse/PED-2987">https://jira.suse.com/browse/PED-2987</a>
</li>
</ul>
</div>